method validation &...

1 © 2012 3M. All Rights Reserved.

METHOD VALIDATION & VERIFICATION APEC-PTIN Project:

CHILE 2013


§  Provide definitions of validation & verification from key resources §  List some key performance characteristics for method validation §  Provide an overview of 3rd party method validation schemes §  Distinguish validation and verification §  List some key performance characteristics for method verification §  Distinguish verification: method verification vs “continuous” verification §  Provide links to references for further use / information

OBJECTIVES


§  CODEX - CAC/GL 74-2010: Guidelines on performance criteria and validation of methods for detection, identification and quantification of specific DNA sequences and specific proteins in foods

§  ISO/DIS 16140-1: Microbiology of food and animal feed, Method validation, Part 1: Vocabulary Currently under revision

§  USDA FSIS: Guidance for Test Kit Manufacturers, Laboratories: Evaluating the Performance of Pathogen Test Kit Methods; and Establishment Guidance for the Selection of a Commercial or Private Microbiological Testing Laboratory

§  US FDA: Guidelines for the Validation of Analytical Methods for the Detection of Microbial Pathogens in Foods

§  Health Canada: The Compendium of Analytical Methods, Volume 1, Development of Methods

§  ISO/IEC 17025:2005 General requirements for the competence of testing and calibration laboratories

Main Validation & Verification Definition Sources:


Reference Validation Definition Codex CAC/GL 741

Process to establish the performance characteristics and limitations of an analytical method: which analytes, in what kind of matrices, in the presence of which interference. Result = precision and trueness values of a certain analytical method under the examined conditions.1 [pg 2]

ISO 16140-12 Establishment of the performance characteristics of a method and provision of objective evidence that the performance requirements for a specified intended use are fulfilled.2 [pg 2]

USDA FSIS3 Process to measure performance characteristics of a particular test, with the goal of determining whether the test is equivalent to the reference test for the intended conditions of use. “Equivalent” = the performance characteristics are statistically indistinguishable.3 [pg 2]

US FDA4 Demonstration that adequate confidence is provided when the results obtained by the alternative method i.e. the commercially available kit, are comparable to or exceed those obtained using the reference method using the statistical criteria contained in the approved validation protocol.4 [pg 16]

Health Canada5 Evaluation of the performance parameters of a new method in comparison to an accepted reference method using paired or unpaired samples. In the context of relative validation, the results of the reference method are assumed to reflect the true microbiological status of the samples and the performance parameters of the alternative method are calculated in relation to this. (Source: Part 4)

ISO 17025:20056 The confirmation by examination and the provision of objective evidence that the particular requirements for a specific intended use are fulfilled.6

WHAT is Validation?


Validation: 1.  Performance characteristics are defined, then… 2.  compared to a reference method, and then… 3.  statistically evaluated to determine “equivalence”

A method is validated for the matrices [& sample sizes] included in the validation

Summary of ALL Validation reference definitions:


Categories of Microbiological Test Methods Performance Characteristics Included in a Validation Study1

Performance Characteristic Quantitative Qualitative (presence/absence)

Relative Accuracy (% Recovery) Yes No

Matrix Effects Yes Yes

Precision Yes No

Sensitivity Yes Yes

Specificity Yes Yes

Inclusivity Yes Yes

Exclusivity Yes Yes

False Positive Rate Yes Yes

False Negative Rate Yes Yes

LOD [POD or LOD50] Yes Yes

LOQ Yes No

Ruggedness Yes Yes

Linearity/Range Yes No

Dr. Michael Brodsky AOAC International 2013 Annual Meeting


Methods requiring validation: ― New or original methods ― Modifications to validated methods ―  Extension of scope to include additional analytes, matrices, or changes in

intended use ― Changes involving new technology or automation ―  Significant parameter changes: reagents/apparatus, time /temperature

incubation periods, enrichment media, etc.

It is possible that small changes/matrix extensions to a related product etc would not

need a ‘full validation’ but may need some level of testing to assess for unintended differences.


§  “Validated” methods are required for conducting “Official Testing” ―  Official = adjective, relating to an authority or public body and its activities and

responsibilities [Oxford online dictionary]

§  European Union regulations specifically stipulate validated methods that have been collaboratively studied

§  Internationally, regulatory agencies require, or at least prefer, to use methods that have been validated and collaboratively studied.

§  Some global regulatory agencies require AOAC Official Methods of Analysis to receive Final Action, before they may be used as an “Official” method

WHY are Validated methods important?


Agency Method Validation Requirements Codex CAC/GL 218

5.2.1 Whenever possible, only methods for which the reliability (accuracy, reproducibility, inter- and intra-laboratory variation) has been statistically established in comparative or collaborative studies in several laboratories should be used.

(EC) No 2073/20079

The use of alternative analytical methods is acceptable when the methods are validated against the reference method in Annex I and if a proprietary method, certified by a third party in accordance with the protocol set out in EN/ISO standard 16140 or other internationally accepted similar protocols, is used. Pg 9

USDA FSIS11 All analytical methods described in FSIS-MLG have been scientifically validated and are considered fit for their intended purpose. Laboratories may also use other validated testing methods that differ from the methods described in FSIS-MLG. For the intended conditions of use, the performance characteristics of the new method and the well characterized method should be statistically indistinguishable Pg 13

US FDA4 FDA-BAM presents the agency's preferred laboratory procedures for microbiological analyses of foods and cosmetics. Methods…fully validated in a collaborative study monitored and evaluated by an independent accrediting body e.g. AOAC-OMA, AFNOR, etc. require in-house verification for the “first use”…

Health Canada5 HPB Methods and Laboratory Procedures are generally used to determine compliance with the Food and Drugs Act and Regulations. In general, methods published in the Compendium of Analytical Methods for which the "application" section is appropriate for the intended purpose are used for microbiological testing of food.

REQUIREMENTS of methods for “Official” use


§ Official Methods = Methods used or accepted for use by a regulatory authority ―  Traditional, cultural methods [may be jurisdiction specific] ―  ISO Method ―  ISO 16140 Validated Methods ―  AOAC Official Methods ―  FDA Bacteriological Analytical Methods ―  USDA FSIS Microbiology Laboratory Methods ―  HC Compendium of Analytical Methods, Volumes 1 to 5

§  Any of the above methods MAY be accepted for use by regulatory agencies, IF: •  They have been compared to an acceptable reference method •  They have been validated as “fit for purpose” for the matrices analyzed

What is considered an OFFICIAL method?


§  ISO ―  AFNOR: French National Organization for Standardization ―  MicroVal: European certification body for microbiology methods ―  NordVal: Nordic certification body: Denmark, Finland, Iceland, Norway and Sweden

§  AOAC Research Institute [RI]: International Standards Organization ―  Performance Tested Method (PTM) ―  Official Method of Analysis (OMA)

§  Health Canada: Canadian certification through Microbiology Methods Committee (MMC) ―  Official Methods (MFOs) ―  HPB Methods (MFHPBs) ―  Laboratory Procedures (MFLPs)

Validation Schemes


AFNOR and MicroVal Certification schemes based on EN/ISO 16140: §  Comparison Study to reference method

1.  Independent expert laboratory co-ordinates validation

2.  Inter-laboratory study [collaborative study]

3.  “All Foods” Claim

§  Regular audit of the quality system of the manufacturer [based on EN 29002 standard (ISO 9002)]

§  Regular verification of the certified methods post certification [every 4 years]


AOAC Research Institute §  AOAC Performance Tested Method [PTM]

―  Pre-collaborative + 1 Independent lab: I/E; Ruggedness testing, paper review of quality system.

―  Yearly renewal – paper process only

―  NO all foods claim – only for matrices evaluated

§  AOAC PTM / Official Method of Analysis [OMA] Harmonization ―  Complete AOAC PTM program / Initiate collaborative study to achieve OMA First Action status;

Final Action status after 2 yrs use in the industry without issue

§  AOAC PTM / OMA/ MicroVal Harmonization

―  Both programs run simultaneously – 1 collaborative study


§ Micro Food Lab Procedure [MFLP] ―  Independent lab [pre-collaborative study] ―  All Foods Claim ―  Reviewed every 5 years ―  Paper review of quality system

§  Health Protection Food Branch [MFHPB] Methods

―  Must be an MFLP method for 1 year prior to collaborative study

Health Canada


§ AFNOR http://www.afnor-validation.com/afnor-validation-validated-methods/validated-methods.html

§ MicroVal http://www.microval.org/validated-methods.html

§ AOAC http://www.aoac.org/iMIS15_Prod/AOAC_Member/Research_Institute/RI_Main.aspx?WebsiteKey=2e25ab5a-1f6d-4d78-a498-19b9763d11b4&hkey=33b744f6-f71e-456a-9305-552469587666&CCO=5

§ Health Canada http://www.hc-sc.gc.ca/fn-an/res-rech/analy-meth/microbio/index-eng.php (Volumes 1 to 5)

§ NordVal http://www.nmkl.org/NordVal/NordValMetoder.htm

Method Scheme Websites – list validated methods including claimed matrices, sample sizes


§  A global Stakeholder panel was formed [through AOAC RI]

§  Performance characteristics between agencies/schemes were compared

§  5 areas for harmonization: 1.  Reference methods – so much variation, some matrix specific 2.  Definition of Fractional Positives – 25-75% positive out of 20 samples 3.  Selection of Food/Category – Chart created for incorporation into ISO 16140-2 4.  Statistical Analysis – RLOD, Chi-Square and/or POD models 5.  Data Sets for Qualitative Collaborative Study/Sample Size – 10 labs / 25 g samples

§  Recent addition: “All Foods” claim now = “Broad range of foods”

Harmonization – ISPAM International Stakeholder Panel on Analytical Methods


Reference Verification Definition Codex Could not find a method verification definition Codex documents [only process verification]

ISO 16140-1 Demonstration that a validated method functions in the user's hands according to the method's specifications determined in the validation study and is fit for purpose

USDA FSIS Laboratory should demonstrate on-going competence in performing the method at their facility, which would include participating in proficiency test programs11 Pg 13

US FDA The confirmation by examination and the provision of objective evidence that specified requirements have been fulfilled.4 Pg 16

Health Canada Verification studies are required when a laboratory implements a fully validated, published or reference standard method. The laboratory should provide evidence, by conducting a secondary evaluation study, that it is capable of achieving the published performance characteristics of the method.5 Vol 1; Part 4; 3.4

ISO 17025:2005 The Laboratory shall confirm that it can properly operate standard methods before introducing the tests or calibrations

Comparison of VERIFICATION definitions


§  Proof that the laboratory can perform the [validated] method as intended

§  FDA has detailed Guidance on in-house verification of a validated method

before use4 Pg 16

Verification for USE

This applies only to matrices and sample sizes that were included in the validation study


§ Validation ― Demonstrate the METHOD is “equivalent” to the reference method [for the

matrices validated] based on defined method criteria

§ Verification ― Demonstrate in the hands of the USER that the method can be performed

to meet the defined method criteria

Distinguishing Validation and Verification


Categories of Microbiological Test Methods Performance Characteristics Included in the Verification of a Validated Method1

Performance Characteristic VALIDATION Quantitative

VALIDATION Qualitative (P/A)

VERIFICATION (Where Applicable)

Relative Accuracy (% Recovery) Yes No Yes Matrix Effects Yes Yes Yes

Precision Yes No Yes

Sensitivity Yes Yes Yes

Specificity Yes Yes Yes

Inclusivity Yes Yes No

Exclusivity Yes Yes No

False Positive Rate Yes Yes Yes

False Negative Rate Yes Yes Yes

LOD [POD or LOD50] Yes Yes No

LOQ Yes No No

Ruggedness Yes Yes No

Linearity/Range Yes No No

Dr. Michael Brodsky AOAC International 2013 Annual Meeting


§  Proficiency Testing: ―  Should be done several times per year ―  Should be handled as a routine sample – not specially handled ―  Can help determine the performance of individual laboratories for specific tests ―  Is used to monitor laboratories’ continuing performance ―  Can demonstrate an analyst’s ability to properly perform the test

§  Test samples: ―  Certified proficiency test providers ―  Internal Ring tests

CONTINOUS Verification - ISO 17025 requirement


§  Definitions of validation & verification §  Key performance characteristics for method validation and verification §  Several global 3rd party method validation schemes were outlined §  Distinguish validation and verification §  References for further use / information

SUMMARY


1.  CODEX - CAC/GL 74-2010 Guidelines on performance criteria and validation of methods for detection, identification and quantification of specific dna sequences and specific proteins in foods http://www.codexalimentarius.org/standards/list-of-standards/

2.  ISO/DIS 16140-1 Microbiology of food and animal feed — Method validation — Part 1: Vocabulary Currently under revision 3.  USDA FSIS: Guidance for Test Kit Manufacturers, Laboratories: Evaluating the Performance of Pathogen Test Kit Methods

http://www.fsis.usda.gov/shared/PDF/Validation_Studies_Pathogen_Detection_Methods.pdf 4.  US FDA: Guidelines for the Validation of Analytical Methods for the Detection of Microbial Pathogens in Foods

http://www.fda.gov/downloads/ScienceResearch/FieldScience/UCM273418.pdf; 5.  Health Canada: Procedure for the Development and Management of Food Microbiological Methods; Part 1. Definitions and Explanations of

Methods and General Procedures; 3. Definition of Terms, 3.1 Method http://www.hc-sc.gc.ca/fn-an/res-rech/analy-meth/microbio/volume1/method_development-eng.php

6.  ISO 17025:2005 General requirements for the competence of testing and calibration laboratories NO LINK, this is copyrighted material 7.  2013 AOAC International Annual Meeting presentation, with permission from: Michael Brodsky, President, Brodsky Consultants, Thornhill, Ontario,

Canada [email protected] 8.  CODEX – CAC/GL 21- 1997 Principles for the Establishment and Application of Microbiological Criteria for Foods

http://www.codexalimentarius.org/standards/list-of-standards/ 9.  COMMISSION REGULATION (EC) No 2073/2005 of 15 November 2005 on microbiological criteria for foodstuffs :

http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=OJ:L:2005:338:0001:0026:EN:PDF 10.  Health Products and Food Branch (HPFB) Standards and Guidelines for Microbiological Safety of Food - An Interpretive Summary 6. Equivalent

Methods http://www.hc-sc.gc.ca/fn-an/res-rech/analy-meth/microbio/volume1/intsum-somexp-eng.php 11.  USDA FSIS: Establishment Guidance for the Selection of a Commercial or Private Microbiological Testing Laboratory:

http://www.fsis.usda.gov/shared/PDF/Guidance_Selecting_Micro_Testing_Lab.pdf 12.  AOAC: AOAC INTERNATIONAL Methods Committee Guidelines for Validation of Microbiological Methods for Food and Environmental Surfaces

http://www.aoac.org/imis15_prod/AOAC_Docs/StandardsDevelopment/AOAC_Validation_Guidelines_for_Food_Microbiology-Prepub_version.pdf

13.  Official Methods for the Microbiological Analysis of Foods http://www.hc-sc.gc.ca/fn-an/res-rech/analy-meth/microbio/volume1/index-eng.php

References

method validation &...

Documents