operating characteristics of frequently used similarity rules

Florian Klinglmueller*

Ack: Andreas Brandt, Thomas Lang, Ina Rondak

Operating characteristics of

frequently used similarity rules

*Austrian Medicines & Medical Devices Agency

The contents of this presentation are my personal opinion.

My remarks do not necessarily reflect the official view of AGES.

Comparison of an originator product to a biosimilar product with respect to

„critical quality attributes“ (i.e. physical, chemical, biological, or microbiological

properties that ensure product quality) with the aim to conclude similarity on the

quality level.

One quality attribute on a continuous scale. Comparison between samples from

originator and biosimilar.

Different rules to decide whether samples from the biosimilar are similar to

samples from the originator - based on sample data

Explore operating characteristics of commonly used rules under different scenarios

of similarity and dissimilarity.

Similarity assessment of quality attributes

Introduction

Decision rule

Problem: Similar is not equal!

⇒ Specify what is similar enough

Average similarity

• E.g.: Biosim on average within ± 1.5

standard deviations of reference mean

• E.g.: Ratio of CQA on average 80%-125%

Population based similarity

• Reference samples define margin of what

is safe (e.g. min-max, TI)

• Future/observed biosim batches fall into

range

Translate into a criterion

How to compute „similar/not similar“ from data

Min-Max: gives a the range of the observed values. Purely descriptive i.e. permits

little inference about future samples of the process, except that true range is wider

X-SD: estimates the variation in the sample around the sample mean. Purely

descriptive, many statistical intervals are constructed by choosing x such that

probabilistic statements hold

• E.g. 95% CI: x=1.96; 95% PI (n=10): x=2.16, 95/95 TI (n=10): x=3.38

Confidence Interval: estimates a range that should cover an unknown parameter

(e.g. mean) of the distribution assumed to generate the data

Prediction Interval: estimates a range that should cover the value of the next

sample from distribution assumed to generate the data

β-content Tolerance Interval: estimates a range that should cover a certain

proportion β of future samples from the distribution assumed to generate the data

Statistical Intervals Probabilistic interpretation of different interval types

Min-Max: gives a the range of the observed values. Purely descriptive i.e. permits

little inference about future samples of the process, except that true range is wider

X-SD: estimates the variation in the sample around the sample mean. Purely

descriptive, many statistical intervals are constructed by choosing x such that

probabilistic statements hold

• E.g. 95% CI: x=1.96; 95% PI (n=10): x=2.16, 95/95 TI (n=10): x=3.38

Confidence Interval: estimates a range that should cover an unknown parameter

(e.g. mean) of the distribution assumed to generate the data

Prediction Interval: estimates a range that should cover the value of the next

sample from distribution assumed to generate the data

β-content Tolerance Interval: estimates a range that should cover a certain

proportion β of future samples from the distribution assumed to generate the data

Statistical Intervals Probabilistic interpretation of different interval types

Frequentist confidence: in repeat experimentation range estimate

computed in this way will cover the quantity (parameter, next sample,

all future samples) a certain proportion of times (e.g. 95%)

Rules for concluding biosimilarity

Min-Max: All samples of the biosim are between min-max of the originator

X-Sigma: All samples from the biosim are within x-standard deviations of the

originators mean

(75%/90%) Tolerance interval: All samples from the biosim are within a P/Q

Tolerance interval of the originator

TI Specs: The P/Q tolerance interval of the biosim is within „specifications“ (e.g.

Min-Max) of the originator

FDA Rule: The 90% confidence interval for mean difference between originator

and biosim is within a similarity margin of 1.5 standard deviations of originator

A selection of frequently used decision rules

Differences in mean, equal variance

Simulation scenario 1

Originator and Biosim samples follow

standard normal distribution

Equal variance

Distance between distributions

expressed as multiples of the

(common) standard deviation

Settings considered:

• M2-M1= {0, 0.5, 0.8, 1, 1.5} * SD

Equal means (biosimilar) shown with dashed lines, unequal means solid lines.

Probability to conclude similarity decreases with increasing dissimilarity (m=10,n=10)

Simulation results: Equal variances Most simple scenario



If we increase the sample size (m=20,n=20) several things happen





Similarity conclusions increase, with increasing originator samples (except TI)





Similarity conclusions increase, with increasing originator samples (except TI)

With increasing Biosim samples similarity conclusions increase for for interval based

criteria


Difference in means, unequal variance

Simulation scenario 2

Originator and biosim distribution may

differ in mean and in variance

Standard deviation of originator is

sdratio times larger than biosim

Values for sdratio: .25 - 2

sdratio=1 corresponds to

Scenario 1

Both cases, assuming equal and

unequal means, were investigated

Impact of different variances on decision rules:

Biosimilar more variable left of horizontal line, reference right of line

Monotone relationship between ratio of variances and probability of concluding

similarity, i.e. more variable reference process -> more likely to conclude similarity

x-Sigma and TI rules often (erroneously) conclude biosimilarity even if Biosim

samples are more variable and means different.

Larger variance in reference to the right

Shift in originator process

Scenario 3

Note: Illustrations use shift of +-5*SD to get a bimodal distribution

Originator samples come from a

mixture of normal distributions with

different means

Means of originator (mixture)

distribution are a multiple of SD apart

Positive shifts are into the direction of

the biosim process

Negative shifts are away from the

biosim process

Impact of shift on decision rules

Dotted line reports probability of self-similarity conclusion

Horizontal line indicates scenario where biosimilar is equal to post-shift process

Probability to conclude similarity is smaller compared to no shift when shift is

slightly opposite to test mean

It increases when shift is towards test mean

However, probability to conclude similarity for acceptance ranges based on

reference SD converges to 1 both for large shifts towards and opposite test mean

Summary & Outlook

Dichotomy of Type I and Type II errors does not apply to an equivalence decision -

boundaries between success and Type I error are fuzzy

Some rules (TI, X-Sigma) have undesirable properties (decreasing power with

increasing sample size, increasing error probability for shifts away from the biosim)

We have only considered simple scenarios; have not considered:

• Alternative designs, use of historical data

• Issues with sampling (originator from the market, biosim from production process under

development)

• Multiplicity (typically there are more than one CQA)

• Sequential decision making (issues with one CQA are discredited using data from another

CQA)

BASG -

Austrian Federal Office for Safety in Health Care

www.basg.gv.at

Traisengasse 5

1200 Vienna

Florian Klinglmueller

Biostatistician

T + 43 (0) 50 555 36624

[email protected]

operating characteristics of frequently used similarity rules

Documents