international seed testing association (ista) the international seed testing association gmo pt...
TRANSCRIPT
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
The International Seed Testing Association GMO PT Program
Design and Analysis
AIEC Fall MeetingOctober 15-16 2014
New Orleans, Louisana
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Outline
• Reference material purity requirements
• Rating labs for ability to detect presence
• Rating labs for ability to quantify
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Reference material purity requirements
Seed Lot Purity = 99%
+ +
- -+ -
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Reference material purity requirements
• To make “sure” that the distributed samples contain the targeted GMO levels– GM seed is required to have purity greater than
99.25% with 95% confidence– Conventional seed is required to have impurity
less than 0.01% with 95% confidence
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Binomial Distribution:
1. Success– With probability p
2. Failure– With probability q = 1-p
How Do I Do That Tim?
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Example 1: Probability of X heads out of 10 coin flips
How many head would you expect to see?
Would you be shocked if all 10 coins came up heads?
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
0 1 2 3 4 5 6 7 8 9 10
0.0
0.05
0.10
0.15
0.20
0.25
Number of Heads
Pro
babi
lity
Coin Tossing: binomial probability of X heads in 10 coin tosses
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
• If I tell you 1% of the beads are red how many would you red beads would you expect to see?
• Would you be surprised if you took a scoop and no red beads were observed?
Example 2: Probability of X red beads out of 300
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
0 1 2 3 4 5 6 7 8 9 10
Positive Seeds
0%
5%
10%
15%
20%
Pro
babi
lity
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
• I have another bag of beads• This bag could have any number of red beads• I let you take 1 and only 1 sample• In the sample you observe no red beads• If you had to say something about the purity
of the new bag what would you say?
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Confidence Claim• If the bag had 1% red beads, a scoop of 300
would contain at least 1 red bead 95% of the time
• I took a scoop of 300 beads and observed 0 red
• Thus, I am 95% confident the percent red beads is less than 1%.
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Using the same logic in our case• To demonstrate GM seed purity greater than
99.25% with 95% confidence requires 400 seeds tested with 0 negatives
• To demonstrate conventional seed impurity less than 0.01% with 95% confidence requires 30,000 seeds tested with 0 positives
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Can we do this cheaper • The cost of testing reference material for PT
program was becoming too large• Needed to find a way to reduce cost and still
make “sure” that the distributed samples contain the targeted GMO levels
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Father of Bayesian Methods
Reverend Thomas Bayes1702 –1761, London England
Nonconformist ministerin Tunbridge Kent, England
and a mathematician on the side
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
• The seed provider has already performed purity tests• By taking into account this information using
Bayesian statistics, the final assessment of seed purity performed by ISTA requires less testing efforts $$$$ savings
Sequential updates:• Prior updated using seed provider’s data• Then update the posterior
1 2 1 2
1 2
1 1 2 1
1 2 1
(chain rule)
( | , ) ( , | ) ( )
( , , ) ( )
( )
( ) ( | ) ( | , )
( | ) ( | , )
f y y f y y f
f y yf
f
f y f y f y y
f y f y y
Prior (given y1) Likelihood (given y1)
: proportionality sign; replaces a constant normalizer
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
• Results :
1 1 11 1 1 1 1( | ) [(1 ) (1 )] [1 (1 ) (1 )]y m yf y Updated prior :
1 1 11 1 1 1
1 1 1 1 1( | ) (1 )[1 (1 ) ] (1 ) 1 (1 )[1 (1 ) ] (1 )y n ym m m mf y
Updated prior :
• Formulas for individual testing (GM seeds):
• Formulas for group testing (conventional seeds):
2 2 22 1 2 2 2 2( | , ) [(1 ) (1 )] [1 (1 ) (1 )]y m yf y y Likelihood :
2 2 22 2 2 22
2 1 2 2 2 22
( | , ) (1 )[1 (1 ) ] (1 ) 1 (1 )[1 (1 ) ] (1 )y n ym m m mn
f y yy
Likelihood :
l1, l2: false negative rates; d1, d2: false positive rates; m1, m2: # of seeds; y1, y2: # of negative seeds observed
l1, l2: false neg. rates; d1, d2: false pos. rates; n1, n2: # of pools ; m1, m2: # of seeds/pool; y1, y2: # of neg. pools observed
1 2 1 2 1( | , ) ( | ) ( | , )f y y f y f y y
. Purity requirements fulfilled. $$$$ savings
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Rating a laboratory in a given proficiency test
• We need to compute objectively, and transparently to the participants, criteria to rate laboratories for each proficiency test on:– Presence/absence – Quantification
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Proficiency tests rating system
• A No problem has been detected in this test
• B There are small problems, but no specific look or action is suggested to the participant
• C Problems, ISTA indicates there might be things to consider by the laboratory to explain or
correct things
• BMP Below Minimum of Performance, ISTA indicates that the results were poor and the
laboratory need to find explanations and to improve/correct
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
One test rating
One test Score Value
Overall rating on 6 tests
Range on 6 tests
A 5 points A 28 – 30 points
B 4 points B 21 – 27 points
C 3 points C 16 – 20 points
BMP 0 points BMP below 16 points
each proficiency test and a run of 6 tests
Example 1:
5A rating and 1 BMP
5*5points + 1*0point =25 points
Overall rating is B
Example 2:
4 B =16 points, no overall BMP
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Rating ability to detect presence
• True value is known• 2 types of error can occur:
– Laboratory report as negative a positive sample
– Laboratory report as positive a negative sample
• The number of mistakes is used as a basis for the rating computations
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Presence/absenceRating system
Rate Percentage of misclassified samples
A 0% - 5%
B >5% - 10%
C >10% - 20%
BMP >20%
Example with 12 samples in a Test
3 mistakes 3/12=25%
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Presence/absenceRating systems
Rate Percentage of misclassified samples
A 0 errors
B 1 error
C 2 errors
BMP More than 2 errors
Example with 12 samples in a Test
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Confidence intervals are rather wide:
If 20 test and all correct 95% confident error rate less than 17%
Many more samples (over 300) would be necessary to estimate error rates with 1% precision
=> PT does not check error rates
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Rating quantification
• True value is known, and/or can be estimated from all results received
• Too many sample results are too far from truth ->BMP• Average results by spiking level are not accurate ->C• Too many inaccurate sample results ->B• Otherwise rate A
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Z-Scores• The rating system for quantification is based
on Z-scores• Before we get to far into the rating system
lets talk really quick about Z-scores
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Example: Birth Weights are Normally Distributed
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
z-scores Consider a value x from a distribution with mean m and standard-deviation s. The formula for converting x into its corresponding z-score is:
x
z
Indicates how far and in what direction x deviates from m , in units of s
z-scores distribution’s mean = 0 z-scores distribution’s std-deviation = 1
When the distribution of reference is normal, the z-scores distribution is also normal and the probability to have a z-score in the interval [-2 ; +2] is ~ 0.95
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
More than ½ of the sample resultsoutside [½ true level; 2 x true level]
C Yes
Probability to observe amore extreme value of the sum
of absolute spiking levels z-scores whenassuming the laboratory provides
accurate resultsis 0.01
No
No
More than 1/6 (~17%) ofthe sample z-scores outside
[-2; +2] (missing sample values are
counted as outside)
BMP Yes
AB Yes No
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
mean of the sample results true levelz_score
Reference Intra Lab std _deviation / # of reported samples by spiking level
Consider the following z-scores computed for each Lab andeach spiking-level:
Probability to observe a moreextreme value of the sum of absolute
spiking levels z-scores when assuming thelaboratory provides accurate results
is 0.01
C Yes No
level by spiking provided samplesof#/estimatedeviation_stdLab_Intra
leveltrueresultssampletheofmeanscore_z
Consider the following z-scores computed for each Lab andeach spiking-level:
The theoretical distribution of the sum of these absolute independentz-scores for a given Lab can be defined using simulations.
C rating will characterize Labs that have problems in providing accuratemean results by spiking levels.
We can the define the rejection region at the 0.01 level:# of spiking levels in the
proficiency test
Rejection region at 0.01
level
1 2.552 3.973 5.254 6.435 7.55…
We can then define the rejection region at the 0.01 level:
# of spiking levels in the
proficiency test
Rejection region at 0.01
level
1 2.552 3.973 5.254 6.435 7.55…
abs(sum(z-scores))
De
nsity
0 2 4 6 8
0.0
0.1
0.2
0.3
3 spiking levels: if sum of absolute z-scores > 5.25 C rating
1%
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
More than 1/6 (~17%) ofthe sample z-scores outside
[-2; +2] (missing sample values are
counted as outside)
AB Yes No
sample result true levelz _ score
Reference Intra Lab std _deviation
Consider the following z-scores computed for each (Lab x spiking-level x sample) combination:
A z-score outside [-2 ; +2] has low probability (~ 0.05) to occur for a laboratory providing accurate sample results.
z-scores
De
nsity
-2 0 2
0.0
0.1
0.2
0.3
0.4
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
. For each Lab k and each spiking level i, compute the variance of the sample results: 2
ik
. Compute the mean of the variances for a given spiking level i (except the one identified with the Cochran’s test): 2
i
Estimate of the intra-Lab standard deviation
. Carry out the Cochran’s test at the 95% level to see if the Lab
with the highest variance has an outlying spread of replicates.
2i. The intra-Lab standard-deviation estimate is then:
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Statistical evaluation of the accuracy of the ISTA GMO PT quantitative rating system
Recent PTs have smaller intra-laboratorystandard-deviations
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Statistical evaluation of the accuracy of the ISTA GMO PT quantitative rating system
Leading to more marginal results for the PTs having smaller intra-laboratorystandard-deviations
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Re
z-fe
scorrenc
sample
e intra
result - true l
-lab std-devie
evel
( )ation
• Estimated from the data• Characterize the precision of the results• The better the precision, the higher chance to get “bad” ratings!
Particularly true for late PTs:
Change implemented (starting PT18):If the relative reference intra-laboratory standard-deviation is lower than 25%, use 0.25 x true GM content as the reference intra-laboratory standard-deviation.
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Conclusion• Reference purity requirements are checked
using Bayesian methods which utilized provider test results
• Labs ability to detect presence/absence rated based on the percent misclassified samples, not intended to estimate error rates
• Labs ability to quantify rated based on z- scores, as labs have gotten more precise some adjustments have needed to be made
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Thanks to:• Jean-Louis Laffont• Kirk Remund
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Thank you for your attention
www.seedtest.o
rg
INTERNATIONAL SEED TESTING ASSOCIATION (ISTA) www.seedtest.org
Questions ?