validation of diagnostic assays oie general principles · pdf filecsiro. oie test development...
TRANSCRIPT
CSIRO.
Axel CollingCSIRO, Australian Animal Health Laboratory3219 Geelong, VictoriaAustralia [email protected]
Second Global Conference of OIE Reference Laboratories and Collaborating CentresOIE Headquarters, Paris, France, 21-23 June 2010
Validation of diagnostic assays OIE general principles and OIE procedure
CSIRO.
< 1990
Sensitivity
Specificity
Repeatability
ReproducibilityPrec
isio
n
Acc
urac
y
1996 2000 2001 2007R. JacobsonPrinciplesof test validation(serology)
M. Greiner & I. Gardner“Fit for purpose”and epidemiological considerations forassay validation
2010
Validation??? WAVLDValidationFromTheory to PracticeMelbourne
2003OIE validation templateand “QualityStandard”
2005
OIE ManualPrinciples & methods& best practice docs
Scotland Design & analysis of diagnostic test evaluationstudies (STARD)
Diagnosis ~ Validation ~ Epidemiology
FAO/IAEAmeetings2002/2003“fit for purpose”standardized a) validationb) recognition c) certification processes
2002
3 OIE ad hoc groupmeetings2008/9 & 2010
2008-9
CSIRO.
Strategy presented at WAVLD 2007, Melbourne….
- The “One-Set-of-Validation-Principles” concept is applicable to all types of diagnostic assays (Ab-Ag, NAD, TSE, etc.)
- One set of Validation Criteria applicable to all assay types (what has to be done?)
- A unified Validation Chapter
- Best Practice documents or annexes, which describe the processes for each assay type (how to do it?)
- A unified template would be accompanied by Unique Guidelines for each assay type
- OIE endorsed this proposal and supported its development through an ad hoc group for test validation
CSIRO.
1) One chapter on test validation, which includes Nucleic Acid Detection Tests (NADs). Before there were two chapters, e.g. one with a focus on serological tests and one for NADs.
2) The chapter includes principles and methods for test validation
3) The chapter has 7 annexes as best practice documents:(The first 3 focus on the development and optimization of diagnostic tests)
- 3.1) Development and optimization of antibody detection assays (Ab) - 3.2) Development and optimization of antigen detection tests (Ag) - 3.3) Development and optimization of nucleic acid detection tests (NAD)
- 3.4) Statistical approaches to test validation including (Latent Class Models) - 3.5) Method comparability (Equivalence testing)- 3.6) Measurement Uncertainty (MU) - 3.7) Selection and use of reference panels
Essential changes in the new approach to international test validation
CSIRO.
4) Guidelines for the use of the OIE validation template
5) “Provisional recognition”Example: Assays developed and used in emergency or outbreak situations. To be assessed: ASe, ASp, repeatability and an estimate for reproducibility
6) “Adjunct tests or procedures”Example: VNT to type an isolated virus or molecular sequencing to confirm a real time PCR result. To be assessed: ASe and ASp.
Essential changes in the new approach to international test validation (cont.)
CSIRO.
Assay Validation’s 3 components
• Assay Development Pathway
• Assay Validation Pathway with its 4 stages
• Validation Retention Pathway
Charting these components…
CSIRO.
OIE test development and validation pathway
Diagnostic Specificity
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples fromreference animals
Candidate test compared withStandard Test Method
Reagents and Controls
Ruggedness
Reproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional Recognition
STAGE 4Implementation
Interpretation oftest results
International recognition (OIE)
Reference standards selected
Deployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Daily in-house QC
Proficiency testing
Replacement of depleted reagents
Assay modifications and re-validation
Equivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panel
Select collaborating Labs
Analytical Sensitivity
Analytical Specificity
Diagnostic Sensitivity
Analytical Specificity
Cut-off determination
Essential Prerequisites
Fitness of Assay for its Intended Purpose
Study Designand
Protocol
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples from experimentalanimals (where used)
Samples fromreference animals
Samples fromreference animals
Candidate test compared withCandidate test compared with
Reagents and ControlsReagents and Controls
RuggednessRuggednessRuggedness
ReproducibilityReproducibilityReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional RecognitionProvisional Recognition
STAGE 4Implementation
Interpretation oftest results
Interpretation oftest results
International recognition (OIE)International recognition (OIE)
Reference standards selectedReference standards selected
Deployment to other LabsDeployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Monitor precisionand accuracy
Daily in-house QCDaily in-house QC
Proficiency testingProficiency testing
Replacement of depleted reagentsReplacement of
depleted reagents
Assay modifications and re-validation
Assay modifications and re-validation
Equivalency assessmentsComparability assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panelDefine evaluation panel
Select collaborating LabsSelect collaborating Labs
Analytical SensitivityAnalytical SensitivityAnalytical Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Diagnostic SensitivityDiagnostic SensitivityDiagnostic Sensitivity
Analytical SpecificityAnalytical SpecificityDiagnostic Specificity
Cut-off determinationCut-off determinationCut-off determination
Essential PrerequisitesEssential Prerequisites
Fitness of Assay for its Intended PurposeFitness of Assay for its Intended PurposeFitness of Assay for its Intended Purpose
CSIRO.
Diagnostic Specificity
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples fromreference animals
Candidate test compared withStandard Test Method
Reagents and Controls
Ruggedness
Reproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional Recognition
STAGE 4Implementation
Interpretation oftest results
International recognition (OIE)
Reference standards selected
Deployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Daily in-house QC
Proficiency testing
Replacement of depleted reagents
Assay modifications and re-validation
Equivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panel
Select collaborating Labs
Analytical Sensitivity
Analytical Specificity
Diagnostic Sensitivity
Analytical Specificity
Cut-off determination
Essential Prerequisites
Fitness of Assay for its Intended Purpose
Study Designand
Protocol
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples from experimentalanimals (where used)
Samples fromreference animals
Samples fromreference animals
Candidate test compared withCandidate test compared with
Reagents and ControlsReagents and Controls
RuggednessRuggednessRuggedness
ReproducibilityReproducibilityReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional RecognitionProvisional Recognition
STAGE 4Implementation
Interpretation oftest results
Interpretation oftest results
International recognition (OIE)International recognition (OIE)
Reference standards selectedReference standards selected
Deployment to other LabsDeployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Monitor precisionand accuracy
Daily in-house QCDaily in-house QC
Proficiency testingProficiency testing
Replacement of depleted reagentsReplacement of
depleted reagents
Assay modifications and re-validation
Assay modifications and re-validation
Equivalency assessmentsEquivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panelDefine evaluation panel
Select collaborating LabsSelect collaborating Labs
Analytical SensitivityAnalytical SensitivityAnalytical Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Diagnostic SensitivityDiagnostic SensitivityDiagnostic Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Cut-off determinationCut-off determinationCut-off determination
Essential PrerequisitesEssential Prerequisites
Fitness of Assay for its Intended PurposeFitness of Assay for its Intended PurposeFitness of Assay for its Intended Purpose
Defining the purpose of the assay guides all subsequent step in the validation process.
Examples of purposes for which an assay may be used:
- Documenting freedom from disease in a country/region- Preventing spread of disease through trade- Eradication of an infection from a region or country- Confirming diagnosis of a clinical case or in a population of animals - Estimating prevalence to facilitate risk analysis- Classifying animals for herd health or immune/vaccination status
Fitness for intended purpose
CSIRO.
Diagnostic Specificity
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples fromreference animals
Candidate test compared withStandard Test Method
Reagents and Controls
Ruggedness
Reproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional Recognition
STAGE 4Implementation
Interpretation oftest results
International recognition (OIE)
Reference standards selected
Deployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Daily in-house QC
Proficiency testing
Replacement of depleted reagents
Assay modifications and re-validation
Equivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panel
Select collaborating Labs
Analytical Sensitivity
Analytical Specificity
Diagnostic Sensitivity
Analytical Specificity
Cut-off determination
Essential Prerequisites
Fitness of Assay for its Intended Purpose
Study Designand
Protocol
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples from experimentalanimals (where used)
Samples fromreference animals
Samples fromreference animals
Candidate test compared withCandidate test compared with
Reagents and ControlsReagents and Controls
RuggednessRuggednessRuggedness
ReproducibilityReproducibilityReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional RecognitionProvisional Recognition
STAGE 4Implementation
Interpretation oftest results
Interpretation oftest results
International recognition (OIE)International recognition (OIE)
Reference standards selectedReference standards selected
Deployment to other LabsDeployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Monitor precisionand accuracy
Daily in-house QCDaily in-house QC
Proficiency testingProficiency testing
Replacement of depleted reagentsReplacement of
depleted reagents
Assay modifications and re-validation
Assay modifications and re-validation
Equivalency assessmentsEquivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panelDefine evaluation panel
Select collaborating LabsSelect collaborating Labs
Analytical SensitivityAnalytical SensitivityAnalytical Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Diagnostic SensitivityDiagnostic SensitivityDiagnostic Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Cut-off determinationCut-off determinationCut-off determination
Essential PrerequisitesEssential Prerequisites
Fitness of Assay for its Intended PurposeFitness of Assay for its Intended PurposeFitness of Assay for its Intended Purpose
Essential prerequisites:
- Reagents availability & costs- Throughput capacity- Turn around time- Technical sophistication- Interpretation skills - QA/QC capability- Requirement for automation?
Essential prerequisites
CSIRO.
Diagnostic Specificity
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples fromreference animals
Candidate test compared withStandard Test Method
Reagents and Controls
Ruggedness
Reproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional Recognition
STAGE 4Implementation
Interpretation oftest results
International recognition (OIE)
Reference standards selected
Deployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Daily in-house QC
Proficiency testing
Replacement of depleted reagents
Assay modifications and re-validation
Equivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panel
Select collaborating Labs
Analytical Sensitivity
Analytical Specificity
Diagnostic Sensitivity
Analytical Specificity
Cut-off determination
Essential Prerequisites
Fitness of Assay for its Intended Purpose
Study Designand
Protocol
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples from experimentalanimals (where used)
Samples fromreference animals
Samples fromreference animals
Candidate test compared withCandidate test compared with
Reagents and ControlsReagents and Controls
RuggednessRuggednessRuggedness
ReproducibilityReproducibilityReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional RecognitionProvisional Recognition
STAGE 4Implementation
Interpretation oftest results
Interpretation oftest results
International recognition (OIE)International recognition (OIE)
Reference standards selectedReference standards selected
Deployment to other LabsDeployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Monitor precisionand accuracy
Daily in-house QCDaily in-house QC
Proficiency testingProficiency testing
Replacement of depleted reagentsReplacement of
depleted reagents
Assay modifications and re-validation
Assay modifications and re-validation
Equivalency assessmentsEquivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panelDefine evaluation panel
Select collaborating LabsSelect collaborating Labs
Analytical SensitivityAnalytical SensitivityAnalytical Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Diagnostic SensitivityDiagnostic SensitivityDiagnostic Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Cut-off determinationCut-off determinationCut-off determination
Essential PrerequisitesEssential Prerequisites
Fitness of Assay for its Intended PurposeFitness of Assay for its Intended PurposeFitness of Assay for its Intended Purpose
Considerable thought and planning is required for study design and protocol (biological, physical and chemical underlaying factors).
- reagents and controls (acquire reagents & instrument testing)- proof of concept experiments or feasibility studies- optimization and standardization (statistician)
(Provided in Best Practices appendices to the Chapter)
Study Design and Protocol
CSIRO.
Diagnostic Specificity
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples fromreference animals
Candidate test compared withStandard Test Method
Reagents and Controls
Ruggedness
Reproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional Recognition
STAGE 4Implementation
Interpretation oftest results
International recognition (OIE)
Reference standards selected
Deployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Daily in-house QC
Proficiency testing
Replacement of depleted reagents
Assay modifications and re-validation
Equivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panel
Select collaborating Labs
Analytical Sensitivity
Analytical Specificity
Diagnostic Sensitivity
Analytical Specificity
Cut-off determination
Essential Prerequisites
Fitness of Assay for its Intended Purpose
Study Designand
Protocol
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples from experimentalanimals (where used)
Samples fromreference animals
Samples fromreference animals
Candidate test compared withCandidate test compared with
Reagents and ControlsReagents and Controls
RuggednessRuggednessRuggedness
ReproducibilityReproducibilityReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional RecognitionProvisional Recognition
STAGE 4Implementation
Interpretation oftest results
Interpretation oftest results
International recognition (OIE)International recognition (OIE)
Reference standards selectedReference standards selected
Deployment to other LabsDeployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Monitor precisionand accuracy
Daily in-house QCDaily in-house QC
Proficiency testingProficiency testing
Replacement of depleted reagentsReplacement of
depleted reagents
Assay modifications and re-validation
Assay modifications and re-validation
Equivalency assessmentsEquivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panelDefine evaluation panel
Select collaborating LabsSelect collaborating Labs
Analytical SensitivityAnalytical SensitivityAnalytical Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Diagnostic SensitivityDiagnostic SensitivityDiagnostic Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Cut-off determinationCut-off determinationCut-off determination
Essential PrerequisitesEssential Prerequisites
Fitness of Assay for its Intended PurposeFitness of Assay for its Intended PurposeFitness of Assay for its Intended Purpose
Optimization is the process by which the most important physical, chemical, and biological parameters of an assay are evaluated and adjusted to ensure that the performance characteristics are best suited to the intended application.
Robustness is a measure of an assay’s capacity to remain unaffected by small changes or variations in test conditions as anticipated in routine laboratory operation.
Lack of robustness may be observed during optimization when critical reagent concentrations are not yet achieved.
(Examples provided in Best Practices appendices to the Chapter)
Optimization & Robustness
CSIRO.
Diagnostic Specificity
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples fromreference animals
Candidate test compared withStandard Test Method
Reagents and Controls
Ruggedness
Reproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional Recognition
STAGE 4Implementation
Interpretation oftest results
International recognition (OIE)
Reference standards selected
Deployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Daily in-house QC
Proficiency testing
Replacement of depleted reagents
Assay modifications and re-validation
Equivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panel
Select collaborating Labs
Analytical Sensitivity
Analytical Specificity
Diagnostic Sensitivity
Analytical Specificity
Cut-off determination
Essential Prerequisites
Fitness of Assay for its Intended Purpose
Study Designand
Protocol
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples from experimentalanimals (where used)
Samples fromreference animals
Samples fromreference animals
Candidate test compared withCandidate test compared with
Reagents and ControlsReagents and Controls
RuggednessRuggednessRuggedness
ReproducibilityReproducibilityReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional RecognitionProvisional Recognition
STAGE 4Implementation
Interpretation oftest results
Interpretation oftest results
International recognition (OIE)International recognition (OIE)
Reference standards selectedReference standards selected
Deployment to other LabsDeployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Monitor precisionand accuracy
Daily in-house QCDaily in-house QC
Proficiency testingProficiency testing
Replacement of depleted reagentsReplacement of
depleted reagents
Assay modifications and re-validation
Assay modifications and re-validation
Equivalency assessmentsEquivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panelDefine evaluation panel
Select collaborating LabsSelect collaborating Labs
Analytical SensitivityAnalytical SensitivityAnalytical Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Diagnostic SensitivityDiagnostic SensitivityDiagnostic Sensitivity
Analytical SpecificityAnalytical SpecificityDiagnostic Specificity
Cut-off determinationCut-off determinationCut-off determination
Essential PrerequisitesEssential Prerequisites
Fitness of Assay for its Intended PurposeFitness of Assay for its Intended PurposeFitness of Assay for its Intended Purpose
Assay Validation Pathway
CSIRO.
Diagnostic Specificity
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples fromreference animals
Candidate test compared withStandard Test Method
Reagents and Controls
Ruggedness
Reproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional Recognition
STAGE 4Implementation
Interpretation oftest results
International recognition (OIE)
Reference standards selected
Deployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Daily in-house QC
Proficiency testing
Replacement of depleted reagents
Assay modifications and re-validation
Equivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panel
Select collaborating Labs
Analytical Sensitivity
Analytical Specificity
Diagnostic Sensitivity
Analytical Specificity
Cut-off determination
Essential Prerequisites
Fitness of Assay for its Intended Purpose
Study Designand
Protocol
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples from experimentalanimals (where used)
Samples fromreference animals
Samples fromreference animals
Candidate test compared withCandidate test compared with
Reagents and ControlsReagents and Controls
RuggednessRuggednessRuggedness
ReproducibilityReproducibilityReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional RecognitionProvisional Recognition
STAGE 4Implementation
Interpretation oftest results
Interpretation oftest results
International recognition (OIE)International recognition (OIE)
Reference standards selectedReference standards selected
Deployment to other LabsDeployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Monitor precisionand accuracy
Daily in-house QCDaily in-house QC
Proficiency testingProficiency testing
Replacement of depleted reagentsReplacement of
depleted reagents
Assay modifications and re-validation
Assay modifications and re-validation
Equivalency assessmentsEquivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panelDefine evaluation panel
Select collaborating LabsSelect collaborating Labs
Analytical SensitivityAnalytical SensitivityAnalytical Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Diagnostic SensitivityDiagnostic SensitivityDiagnostic Sensitivity
Analytical SpecificityAnalytical SpecificityDiagnostic Specificity
Cut-off determinationCut-off determinationCut-off determination
Essential PrerequisitesEssential Prerequisites
Fitness of Assay for its Intended PurposeFitness of Assay for its Intended PurposeFitness of Assay for its Intended Purpose
Assay Validation Pathway
Repeatability is the level of agreement between results of replicates of a sample both within and between runs of the same test method in a given laboratory.
Analytical specificity is the degree to which the assay distinguishes between the target analyte and other components in the sample matrix.
Analytical sensitivity is the smallest detectable amount of analyte in a sample that can be measured with a defined certainty.
CSIRO.
Diagnostic Specificity
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples fromreference animals
Candidate test compared withStandard Test Method
Reagents and Controls
Ruggedness
Reproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional Recognition
STAGE 4Implementation
Interpretation oftest results
International recognition (OIE)
Reference standards selected
Deployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Daily in-house QC
Proficiency testing
Replacement of depleted reagents
Assay modifications and re-validation
Equivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panel
Select collaborating Labs
Analytical Sensitivity
Analytical Specificity
Diagnostic Sensitivity
Analytical Specificity
Cut-off determination
Essential Prerequisites
Fitness of Assay for its Intended Purpose
Study Designand
Protocol
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples from experimentalanimals (where used)
Samples fromreference animals
Samples fromreference animals
Candidate test compared withCandidate test compared with
Reagents and ControlsReagents and Controls
RuggednessRuggednessRuggedness
ReproducibilityReproducibilityReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional RecognitionProvisional Recognition
STAGE 4Implementation
Interpretation oftest results
Interpretation oftest results
International recognition (OIE)International recognition (OIE)
Reference standards selectedReference standards selected
Deployment to other LabsDeployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Monitor precisionand accuracy
Daily in-house QCDaily in-house QC
Proficiency testingProficiency testing
Replacement of depleted reagentsReplacement of
depleted reagents
Assay modifications and re-validation
Assay modifications and re-validation
Equivalency assessmentsEquivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panelDefine evaluation panel
Select collaborating LabsSelect collaborating Labs
Analytical SensitivityAnalytical SensitivityAnalytical Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Diagnostic SensitivityDiagnostic SensitivityDiagnostic Sensitivity
Analytical SpecificityAnalytical SpecificityDiagnostic Specificity
Cut-off determinationCut-off determinationCut-off determination
Essential PrerequisitesEssential Prerequisites
Fitness of Assay for its Intended PurposeFitness of Assay for its Intended PurposeFitness of Assay for its Intended Purpose
Assay Validation Pathway
Diagnostic specificity is the proportion of known uninfected reference animals that test negative.
Diagnostic sensitivity is the proportion of known infected reference animals that test positive in an assay.
Cut-off and threshold are considered to be synonymous. Intermediate, indeterminate, suspicious or equivocal are terms used for a zone of test values between the positive and negative cut-offs.
CSIRO.
Diagnostic Specificity
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples fromreference animals
Candidate test compared withStandard Test Method
Reagents and Controls
Ruggedness
Reproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional Recognition
STAGE 4Implementation
Interpretation oftest results
International recognition (OIE)
Reference standards selected
Deployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Daily in-house QC
Proficiency testing
Replacement of depleted reagents
Assay modifications and re-validation
Equivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panel
Select collaborating Labs
Analytical Sensitivity
Analytical Specificity
Diagnostic Sensitivity
Analytical Specificity
Cut-off determination
Essential Prerequisites
Fitness of Assay for its Intended Purpose
Study Designand
Protocol
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples from experimentalanimals (where used)
Samples fromreference animals
Samples fromreference animals
Candidate test compared withCandidate test compared with
Reagents and ControlsReagents and Controls
RuggednessRuggednessRuggedness
ReproducibilityReproducibilityReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional RecognitionProvisional Recognition
STAGE 4Implementation
Interpretation oftest results
Interpretation oftest results
International recognition (OIE)International recognition (OIE)
Reference standards selectedReference standards selected
Deployment to other LabsDeployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Monitor precisionand accuracy
Daily in-house QCDaily in-house QC
Proficiency testingProficiency testing
Replacement of depleted reagentsReplacement of
depleted reagents
Assay modifications and re-validation
Assay modifications and re-validation
Equivalency assessmentsEquivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panelDefine evaluation panel
Select collaborating LabsSelect collaborating Labs
Analytical SensitivityAnalytical SensitivityAnalytical Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Diagnostic SensitivityDiagnostic SensitivityDiagnostic Sensitivity
Analytical SpecificityAnalytical SpecificityDiagnostic Specificity
Cut-off determinationCut-off determinationCut-off determination
Essential PrerequisitesEssential Prerequisites
Fitness of Assay for its Intended PurposeFitness of Assay for its Intended PurposeFitness of Assay for its Intended Purpose
Assay Validation Pathway
Reproducibility is the ability of a test method to provide consistent results when applied to aliquots of the same samples tested at different laboratories.
Ruggedness is a measure of the assay’s capacity to remain unaffected by substantial changes or substitutions in test conditions anticipated in multi-laboratory utilisation, part of fitness studies and reproducibility assessments (e.g. impact of shipping conditions, technology transfer, reagents batches, equipment, testing platforms and/or environments).
CSIRO.
Diagnostic Specificity
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples fromreference animals
Candidate test compared withStandard Test Method
Reagents and Controls
Ruggedness
Reproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional Recognition
STAGE 4Implementation
Interpretation oftest results
International recognition (OIE)
Reference standards selected
Deployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Daily in-house QC
Proficiency testing
Replacement of depleted reagents
Assay modifications and re-validation
Equivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panel
Select collaborating Labs
Analytical Sensitivity
Analytical Specificity
Diagnostic Sensitivity
Analytical Specificity
Cut-off determination
Essential Prerequisites
Fitness of Assay for its Intended Purpose
Study Designand
Protocol
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples from experimentalanimals (where used)
Samples fromreference animals
Samples fromreference animals
Candidate test compared withCandidate test compared with
Reagents and ControlsReagents and Controls
RuggednessRuggednessRuggedness
ReproducibilityReproducibilityReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional RecognitionProvisional Recognition
STAGE 4Implementation
Interpretation oftest results
Interpretation oftest results
International recognition (OIE)International recognition (OIE)
Reference standards selectedReference standards selected
Deployment to other LabsDeployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Monitor precisionand accuracy
Daily in-house QCDaily in-house QC
Proficiency testingProficiency testing
Replacement of depleted reagentsReplacement of
depleted reagents
Assay modifications and re-validation
Assay modifications and re-validation
Equivalency assessmentsEquivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panelDefine evaluation panel
Select collaborating LabsSelect collaborating Labs
Analytical SensitivityAnalytical SensitivityAnalytical Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Diagnostic SensitivityDiagnostic SensitivityDiagnostic Sensitivity
Analytical SpecificityAnalytical SpecificityDiagnostic Specificity
Cut-off determinationCut-off determinationCut-off determination
Essential PrerequisitesEssential Prerequisites
Fitness of Assay for its Intended PurposeFitness of Assay for its Intended PurposeFitness of Assay for its Intended Purpose
Assay Validation Pathway
http://www.oie.int/vcda/eng/VCDA_dossier.dot
Application Form for the Certification of Diagnostic Tests as validated fit for specific purposes
CSIRO.
Diagnostic Specificity
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples fromreference animals
Candidate test compared withStandard Test Method
Reagents and Controls
Ruggedness
Reproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional Recognition
STAGE 4Implementation
Interpretation oftest results
International recognition (OIE)
Reference standards selected
Deployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Daily in-house QC
Proficiency testing
Replacement of depleted reagents
Assay modifications and re-validation
Equivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panel
Select collaborating Labs
Analytical Sensitivity
Analytical Specificity
Diagnostic Sensitivity
Analytical Specificity
Cut-off determination
Essential Prerequisites
Fitness of Assay for its Intended Purpose
Study Designand
Protocol
Study Designand
Protocol
Samples from experimentalanimals (where used)
Samples from experimentalanimals (where used)
Samples fromreference animals
Samples fromreference animals
Candidate test compared withCandidate test compared with
Reagents and ControlsReagents and Controls
RuggednessRuggednessRuggedness
ReproducibilityReproducibilityReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
Repeatability and preliminaryReproducibility
STAGE 2Diagnostic
characteristics
Assay Validation Pathway
STAGE 1
Analyticalcharacteristics
Provisional RecognitionProvisional Recognition
STAGE 4Implementation
Interpretation oftest results
Interpretation oftest results
International recognition (OIE)International recognition (OIE)
Reference standards selectedReference standards selected
Deployment to other LabsDeployment to other Labs
Validationstatus
retention
Monitoring andmaintenanceof validation
criteria
Monitor precisionand accuracy
Monitor precisionand accuracy
Daily in-house QCDaily in-house QC
Proficiency testingProficiency testing
Replacement of depleted reagentsReplacement of
depleted reagents
Assay modifications and re-validation
Assay modifications and re-validation
Equivalency assessmentsEquivalency assessments
AssayDevelopment
Pathway Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
Optimization, Robustness, Calibration to Standards
STAGE 3ReproducibilityDefine evaluation panelDefine evaluation panel
Select collaborating LabsSelect collaborating Labs
Analytical SensitivityAnalytical SensitivityAnalytical Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Diagnostic SensitivityDiagnostic SensitivityDiagnostic Sensitivity
Analytical SpecificityAnalytical SpecificityAnalytical Specificity
Cut-off determinationCut-off determinationCut-off determination
Essential PrerequisitesEssential Prerequisites
Fitness of Assay for its Intended PurposeFitness of Assay for its Intended PurposeFitness of Assay for its Intended Purpose
- Proper maintenance and upgrading of the assay requires an “on going validation”
Validation Status Retention
CSIRO.
1) Fitness for intended purpose(s)2) Optimisation3) Standardisation4) Robustness5) Repeatability 6) Analytical sensitivity7) Analytical specificity8) Thresholds (cut-offs)9) Diagnostic sensitivity10) Diagnostic specificity11) Reproducibility12) Ruggedness
12 criteria that must be addressed in the development and validation of diagnostic tests
CSIRO.
The new proposed OIE validation process
OIE validation navigation flowchart
Chapter 1.1.4. Principles and methods of validation of diagnostic assays for infectious diseases
OIE validation template
1.1.4.1.Development andoptimizationof Ab detection tests
1.1.4.2.Development andoptimizationof Ag detection tests by direct+ immunolo-gical means
1.1.4.3.Development andoptimizationof NucleicAcid Detectiontests (NAD)
1.1.4.4.Statisticalapproaches to validation
1.1.4.5.Methodcomparability
1.1.4.6.MeasurementofUncertainty(MU)
1.1.4.7.Selection and use of referencepanels
7 best practice documents or annexes
Guidelinesfor Immunological based assays, e.g. ELISA, CFT, Luminex,Ag, Ab.
Guidelinesfor NADs
Guidelinesfor TSEagentdetectionassays
3 guidelines documents
CSIRO.
- We have come a long way toward a strong science based assay development and validation model.
- Validation is a process that determines the fitness of an assay, which has been properly developed, optimised and standardised, for an intended purpose. Validation includes estimates of the analytical and diagnostic performance characteristics of a test.
- We are close to implementing detailed processes that will harmonize validation efforts and drive assay validation science toward excellence.
- FAO/IAEA/OIE have given green light for a consultants meeting in Vienna, 6-9 September 2010 to develop a roadmap for the implementation of modern OIE principles and methods for diagnostic test validation.
Conclusions
CSIRO.
- OIE ad hoc group for Test Validation and OIE
- Working group for test development and validationof Subcommittee for Animal Health Laboratory Standards Australia and New Zealand (SCAHLS)Joseph O’Keefe
- National Animal Health Laboratory Network, NAHLNBarbara Martin
- FAO/IAEA, Gerrit Viljoen, Adama Diallo
- Australian Animal Health Laboratory, AAHL John White, Ross Lunt Wilna Voslo, Jianning Wang Martyn Jeggo, Peter Daniels
Rear f.l.t.r. Peter Wright, Canada, Richard Jacobson, USAFront f.l.t.r.: Axel Colling, Australia, Ian Gardner, USA, Francois Diaz, OIEApologies: Katherine Webster, UK, Sandor Belak, Sweden
Acknowledgements