method validation and verification: an overview patricia hanson, biological administrator i florida...
TRANSCRIPT
Method Validation and Verification: An OverviewPatricia Hanson, Biological Administrator I
Florida Department of Agriculture and Consumer Services, Food Safety Microbiology
Laboratory
Introduction• 17 years in the microbiology section of the
Florida Department of Agriculture and Consumer Services, Food Laboratory
• Laboratory moved from primarily cultural methods in 1998 to a leader in the implementation of new technology in 2015
• Numerous partnerships with state, federal and industrial customers to validate and implement new methods
Goals• Distinguish between method verification and
method validation• Based on our laboratory’s experience, identify
the critical components of both a method verification and a method validation
• Offer insight to approaches we have found useful, specifically in the microbiology section
• Share lessons learned
Verification vs Validation• Verification is a demonstration that your
laboratory can perform a Standard Method or other well documented method and produce acceptable results
• Validation is both a demonstration that your lab can run the method and that the method or change to a method is fit for purpose
Aspects of a Verification• Run entire method, start to finish, exactly as it
is written• Include both positive and negative samples• Represent all matrix types• Be spiked at a level that is fit for the purpose
of the method if spiked samples are used
Verifications can be ongoing….• For the purpose of this discussion,
“Verification” is being used to describe what the laboratory does prior to implementing a method in the laboratory
• The term “Verification” is often also used to describe the ongoing process of verifying a method with each run – in our lab this it the use of Process Controls, Cultural Controls, and Media Controls
Validation• Laboratory developed method
• Matrix Extensions
• ANY technically significant change to a Standard Method or previously validated method
Overview of an Existing Guideline
• Level One: Urgent Use, Single Lab Validation
• Level Two: Single Lab Validation
• Level Three: Multi-Lab Validation
• Level Four: Full Collaborative Study
Level 2: Single Lab Validation Highlights
• 50 different strains of target organism, if available
• 30 different strains of “nearest neighbor” organisms
• At least two target analyte levels
• 20 replicates per food type per analyte level
• Perform side by side if possible
Aspect of a Verification• Be extensive enough to show that the method
is fit for the intended use and meets the customer’s needs
What is “Extensive Enough”Items to be considered:Matrix: number and typesTarget: source and levelsMethod Evaluation: side by side or use of
standard reference material When considering these items during the
development of the validation, keep in mind what is needed to show the method is fit for the intended use and meets the customer needs.
Matrices• Validation or verification needs to be
performed on each food type• Meat (raw, smoked, cooked, etc)• Vegetable (whole, cut, sprout, etc)• Dairy (liquid, solid, etc)• Other (spices, candies, etc)
Replicates• Exact number depends on method• Verification, normally triplicate• Validation, normally six replicates per level• Important to do the same number of
replicates of “un-spiked”
Naturally Contaminated Samples• Naturally contaminated pros and cons
• Best test of the method
• Level of analyte is unknown
• Little or no control of analyte level
Spiked Samples• Spiked sample pros and cons• Ability to control organism level• Methods often perform differently on spiked
samples than on naturally contaminated samples
Aging of Spiked Cultures• Spiking samples 48-72 hours prior to initiation
of testing can better represent naturally occurring contamination
• Sometimes the cells are so stressed, they die off
• Consider the method or part of method being validated and type of validation
Selection of Organisms• Select at least one organism for each target
• “Nearest Neighbor” organisms – not often used but good to consider
“Side by Side” Validations• Normally involves running the method being
validated side by side with a Standard or established method
• Twice the work but very valuable for determining if the “new” method is equivalent or better than the Standard or established method
• Best choice when changing part of a method – for example a new enrichment broth but same screening method
How Much Confirmation?• Screening Method Validations: Confirm
everything• You can get by with an abbreviated
confirmation, for example just biochemical panel and/or serology but do enough to show you “got out what you put in.”
Why Do So Many Confirmations?• Ensure you can confirm samples that screen
positive
• Ensure samples you can confirm screen positive
• Spiked samples that don’t screen positive and don’t confirm point to limitations of the method or problems with the spiking procedure
Data Evaluation• The measures used to evaluate data depend on
the purpose and type of the method and the needs of the customer
Accuracy
RUGGEDNESSLimit
of
Quantitatio
n
SPECIFICITY
False Positive RatePrecision
Range
Qualitative Data Evaluation• Detection Limits – can the method detect the target
at the levels required by the customer?• Specificity – can the method detect the target in the
matrix?• Selectivity – can the method differentiate between
the target and other analytes in the matrix?• Robustness – can the method be repeated on the
same sample with the same results?
Quantitative Data Evaluation• Detection Limits, Specificity, Selectivity, and
Robustness• Precision – Does the method produce the
same results as established method?• Linearity – Does the method produce
proportional results with in the customer’s target range?
Personnel Considerations• Personnel designing and approving verifications
and validations should be knowledgeable about the technology as well as the needs of the lab
Personnel Considerations• Personnel carrying out verification and
validations should be competent in all techniques
• Work on verification and validation studies can be used to demonstrate competency but should not be used as training
Pit-falls of Method Validation• Repeating tests until you get the result you
want • Drifting from method validation into method
development• Adjusting criteria for acceptance AFTER the data is obtained
Overcoming Pit-falls• Have a well thought out, documented and
approved plan and stick to it
• Go back and start over, incorporating any changes into new validation
• All data should still be included in evaluation
Documentation
• Approval of plan prior to running• Raw Data• Interpretation of raw data • Final approval from customer• Reference validation in SOP• If SOP is a modified Standard Method, list
deviations from the Standard Method in a section in the SOP
References• U. S. Department of Health and Human Services, Food and
Drug Administration (2014, August 29). Volume II – Methods, Method Verification and Validation ORA-LAB.5.4.5. Retrieved from http://www.fda.gov/ScienceResearch/FieldScience/ucm171877.htm
• U. S. Department of Health and Human Services, Food and Drug Administration (2011, September 8). Guidelines for the Validation of Analytical Methods for the Detection of Microbial Pathogens in Foods. Retrieved from http://www.fda.gov/downloads/ScienceResearch/FieldScience/UCM273418.pdf