validation: concept, & considerations
DESCRIPTION
Validation: concept, & considerations. Beni Kaufman. Will be presenting:. Review The concept Validation Components and their measurement experimental design of PCR validation Process vs. Modular validation. References:. Guidance for Industry: Bioanalytical Method Validation. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/1.jpg)
Validation: concept, & considerations
Beni Kaufman
![Page 2: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/2.jpg)
Will be presenting:
• Review– The concept – Validation Components and
their measurement– experimental design of PCR
validation
• Process vs. Modular validation
![Page 3: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/3.jpg)
References:
• Guidance for Industry: Bioanalytical Method Validation. U.S. Department of Health and Human Services, Food and Drug Administration (FDA), Center for Drug Evaluation and Research (CDER), Center for Veterinary Medicine (CVM) May 2001
• PCR Validation & Performance Characteristics Analytical Environmental Immunochemical Consortium (AEIC) Biotech Consensus Paper; S. Charlton, R. Giroux, D. Hondred, C. Lipton, K. Worden
• Validation of Analytical procedures: Methodology, International Conference on Harmonization of Technical Requirements for the Registration of Pharmaceuticals for Human Use, 1996
![Page 4: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/4.jpg)
Warning
Politically sensitive material!
Politically!!!
sensitive material!!
Discuses components… avoid criteria!
![Page 5: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/5.jpg)
Validation Components
![Page 6: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/6.jpg)
Selectivity (Specificity)
• The ability of the analytical method to differentiate (and quantify) the analyte in the presence of other components in the sample (to amplify only the Sequence of interest.)
Selectivity may be affected by:– Interference:
• Cross amplification of non target sequences (function of, Primer design)
– Matrix effects: • Background signal (Sybr green)• Quality & quantity of DNA• Reaction conditions (master-mix,
thermocycling profile)
![Page 7: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/7.jpg)
Selectivity (Cont.)
Assessed by: – Fragment length analysis (right size
amplicon)• Electrophoresis gel analysis• CE
![Page 8: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/8.jpg)
Dissociation Curvedo not use r2774, 02-08-2006, 15Hr 58Min.mxp
Assessed by:
–Melting curve analysis
Selectivity (Cont.)
![Page 9: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/9.jpg)
Precision• The closeness of agreement (degree
of scatter) between a series of measurements obtained from multiple sampling of the same homogeneous sample under the prescribed conditions.
• Variation among rep.s within an assay
• Same as Repeatability
• Measured by parameters of variation, mostly %CV
![Page 10: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/10.jpg)
Precision parameters:
Sample Result Average VarianceStandard Deviation %CV
=AVERAGE(D5:D8) =VAR(D5:D8)
=STDEV(D5:D8)
=100*(StDev/Average)
1.1 0.12 0.1475 0.00383 0.061847 41.929888
1.2 0.24
1.3 0.11
1.4 0.12
2.1 0.3 0.31 0.00047 0.021602 6.9685384
2.2 0.34
2.3 0.29
2.4 0.31
3.1 0.52 0.48 0.003 0.054772 11.410887
3.2 0.51
3.3 0.49
3.4 0.4
![Page 11: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/11.jpg)
Accuracy/Trueness
• The closeness of mean test results to the true value of the analyte.
Qualitative assay:Measured by error rate:% false positive = False
positives/ # of negatives% false negative = False
negatives/# of positives
![Page 12: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/12.jpg)
Accuracy/Trueness (cont.)
• Quantitative assay:– The mean recovery at several
points across the quantitative range
% Recovery =100 (observed/actual)
(also,
the deviation of the mean from the true value)
![Page 13: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/13.jpg)
Accuracy/Trueness measured
Truelevel (%) Sample Result %Recovery
Average Recovery
0.1 sample 1 0.12 120 147.5
0.1 sample 2 0.24 240
0.1 sample 3 0.11 110
0.1 sample 4 0.12 120
0.3 sample 1 0.3 100 103.3333
0.3 sample 2 0.34 113.3333333
0.3 sample 3 0.29 96.66666667
0.3 sample 4 0.31 103.3333333
0.5 sample 1 0.52 104 96
0.5 sample 2 0.51 102
0.5 sample 3 0.49 98
0.5 sample 4 0.4 80
![Page 14: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/14.jpg)
Linearity & Range• Linearity: The ability of the
assay (within a given range) to obtain test results which are directly proportional to the concentration/amount of the analyte
• Range: The interval between the upper & lower concentrations of an analyte for which the assay has suitable levels of precision, accuracy & linearity.
![Page 15: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/15.jpg)
Linearity & Range (cont.)
• Linearity and Range can be evaluated simultaneously
• Demonstrated on a dilution series (transgene genomic DNA/null genomic DNA) across a relevant range of concentrations
• The Range is established by confirming acceptable degrees of linearity, accuracy, & precision, within or at the extremes of a specified range.
![Page 16: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/16.jpg)
Linearity evaluated
• Linearity is evaluated by a plot of signals as a function of analyte concentration & linear regression analysis.
![Page 17: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/17.jpg)
![Page 18: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/18.jpg)
Sensitivity
Two concepts of sensitivity:1. Change in response per amount
of reactant -> dose-response curve
In PCR the dose response is derived from the amplification efficiency - We optimize the assay for a maximal dose response (~100% amp. Efficiency)
Therefore,– dose-response is reflected in the
standard curve. It’s captured in the Linearity component & • is the basis for quantification• The source of our resolution power
![Page 19: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/19.jpg)
Sensitivity (cont.)
2. Limit of detection (LOD), The minimum amount of target analyte that can be detected with a given level of confidence• Applies to QL & QT PCR
Limit of quantification (LOQ), The lowest amount of target analyte that can be quantified with acceptable levels of precision and accuracy.
• Applies only to QT PCR
![Page 20: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/20.jpg)
Determining LOD & LOQ:
“Spiking” series:• Decreasing amounts of transgenic
seed are mixed in with conventional seed to create a series of seed pools with varying proportion of transgenes.
• Seed pools are ground to flour
• DNA isolated from flour and used for PCR; targeting the corresponding target sequence.
![Page 21: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/21.jpg)
Sensitivity (cont.)
The LOD will be lowest spike detected with an acceptable confidence level.
The LOQ will be the lowest spike that can be differentiated from zero with an acceptable confidence level
![Page 22: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/22.jpg)
Ruggedness
• The effectiveness of an analytical process in face of small environmental/operating conditions, such as:– Different analysts– Different equipment– Different labs
• Effectiveness is measured as changes in the precision or accuracy.
![Page 23: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/23.jpg)
Ruggedness (cont.)Effectiveness is measured as changes in the
precision or accuracy:• For qualitative PCR evaluated by the changes in
error rate and LOD• For quantitative PCR evaluated by HORRAT
Where the Relative Standard Deviation of Reproducibility (RSDr) is given as:
RSDr = 2(1-0.5lnC) ~ 2C-0.1505
(C= concentration or quantity) AndHORRAT =
RSDr(observed)/RSDr(expected) HORRAT is expected to be close to 1
• Horwitz, W. (1995) Protocol for the design, conduct and interpretation of method performance studies, Pure and Appl. Chem, 67:331-343
![Page 24: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/24.jpg)
Ruggedness measured
Spike (%)
Result (%) RSDr Obs RSDr Exp HORRAT
=2(1-0.5lnResult) =2(1-0.5lnTrue) =RSDr obs/RSDr exp
0.1 0.1475 0.086 0.303 0.283828383
0.3 0.31 0.829 0.796 1.041457286
0.5 0.48 1.266 1.307 0.968630451
0.6 0.6375 2.775 1.489 1.863666891
1 1.15 2.14 2 1.07
1.5 1.65 2.501 2.405 1.03991684
2.2 2.55 2.936 2.788 1.053084648
1.045797786
![Page 25: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/25.jpg)
Robustness
• Describes the reliability of an analysis with respect to variations in method parameters.
• Measured by experimentally defining the critical range of:– Template concentration– Primer concentration– Mg2 Concentration– Thermocycling temperature range
Usually part of the assay optimization, prior to the
validation process.
![Page 26: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/26.jpg)
Cartoon Break
![Page 27: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/27.jpg)
Seems to be a tedious process!
It
Is !!!
![Page 28: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/28.jpg)
But,
the right experimental design
Can take away some of the edge…
For example:
![Page 29: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/29.jpg)
QT PCR Validation design:
Experiment:• Series of conventional seed pools
fortified with transgenic seed at a decreasing ratio.(For example: from 2% to 0.01% at -0.5X increments).
• Highest level serves as positive control • Negative control• Five reps per level • Isolate, quantify, normalize, PCR (IQNP)• All in all: 8 spike levels x 5 replicates =
40 amplifications• Repeat 3 times, 3 different
instruments, different analysts, (3 different dates (?) Astrological effect)
![Page 30: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/30.jpg)
QT PCR Validation design:
Analyze • Selectivity: all amplifications yielded the
right size amplicon (on gel, or by Tm)• Precision: Calculate %CV among reps
within plates• Accuracy: Calculate mean % recovery
within plates• Linearity: use samples as standards –
create standard curve- test linearity• Range: based on results of Precision,
Accuracy, & Linearity; define range.• LOD: Identify the lowest detected spike
with an accepted confidence limit • LOQ: Identify lowest spike that its
confidence interval does not overlap zero.
• Ruggedness: HORRAT, or alternatively, ANOVA between plates, runs, annalists.
![Page 31: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/31.jpg)
QL PCR Validation design
Experiment• Series of conventional seed pools
fortified with transgenic seed at a decreasing ratio.
(… from 2% to 0.01% at -0.5X increments).
• Highest level serves as positive control • Negative control• Five reps per level • IQNP
Analyze:• Selectivity: all amplifications yielded
the right size amplicon (On gel or by Tm)
• LOD: The lowest spike level to yield amplification = tentative LOD
![Page 32: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/32.jpg)
QL PCR Validation design
Experiment:• Two plates, each plate, half null,
and half spiked at tentative LOD. • Isolate, quantify, normalize, • PCR the two plates on different
instruments, different analysts, etcAnalyze:• Accuracy: Calculate positive and
negative error rate. • Confirm LOD: if %false negative
< defined criteria (5?)• Ruggedness: compare error rates
between plates/instruments/analysts
![Page 33: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/33.jpg)
That wasn’t that bad wasn’t it?
![Page 34: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/34.jpg)
Not only PCR!The testing process is made of a
number of consecutive steps, all can be validated, some have to be validated
• Sampling• Sub-sampling• DNA Isolation• DNA Quantification • DNA Normalization• PCR• Post-PCR• Data Analysis
![Page 35: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/35.jpg)
Modular Validation
The recognition that many of the applications – steps, in the testing process require independent validation of their function
&
For better efficiency
Brought about the idea of Modular Validation
A. Holst-Jensen, J-AOAC, 1995
![Page 36: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/36.jpg)
Modular Validation
Validate each step (module).
Once, validated, different modules can be combined in to a process that no longer
require validation
DNA is DNA!?
IT IS NOT.
![Page 37: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/37.jpg)
Whole Process Validation
– Particle size– DNA isolation efficiency– Instrument error– Matrix effect– Standards
All affect the out come of the testing process, therefore, the validation is of the whole process and only in the context of the given matrix, instrumentation, & standards…
![Page 38: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/38.jpg)
You can’t “mix & match”
Any deviation…will require
VALIDATION.
![Page 39: Validation: concept, & considerations](https://reader036.vdocuments.mx/reader036/viewer/2022062500/568158d6550346895dc61d45/html5/thumbnails/39.jpg)