David said, ”Reginald Reagan suggests that the statistician should startwith distribution strongly peaked near 1/2, because why else would the drugcompany use this protocol. Maybe that’s right, but I have the vague feeling thatthere are game theoretic issues associated to updating your views in responseto something an adversary says.”

I should clarify. First of all, I am not suggesting you update on the drugcompany’s protocol. I’m assuming that they don’t have any more informationabout the drug than we do. If I thought they did know something more aboutthe drug, then I would update on their protocol, as you said, but that’s kind ofleaving behind the thought experiment.

What I’m saying is, if we are going to use a prior distribution that reflectsour prior beliefs, then we end up in one of two situations:

1. We don’t think the drug company can make us believe θ > 1/2

2. Our prior dist for θ is strongly peaked near 1/2

The significance of this is that, in neither case does the OP convince us notto do the thought experiment.

1 Uniform prior belief, and a uniform prior dis-tribution

Our prior is, what, uniform? So we think all values of θ are possible. Myassertion is that, if this is really our prior belief, we don’t think that we’ll betricked. I’ll show this by looking at the probability their stopping rule appliesat θ < 1/2.

Let SN be the number of positive observations after testing N people. SN

has a binomial distribution with parameters θ and N .The probability that the drug company stops is the probability that, for

some N from 1 to infinity, SN > N/2 +√N . That is,

P

∞∨i=dN/2+

√Ne

SN >N

2+√N

By Boole’s inequality, this is less than

∞∑i=dN/2+

√Ne

P

(SN >

N

2+√N

)

And these probabilities we calculate with the binomial distribution of SN men-tioned above.

This bound gets low fast as you leave .5:

1

> term <- function(t, N) pbinom(N/2+sqrt(N), N, t, lower.tail=FALSE)

> bound <- Vectorize(function(t) sum(term(t, 1:10^5)))

> plot(bound, xlim=c(0,1), ylim=c(0,1),

+ xlab='True success probability theta', ylab='bound',+ main='Upper bound on probability of ever stopping')

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Upper bound on probability of ever stopping

True success probability theta

boun

d

So, imagine us reviewing the protocol, deciding how we’re going to analyzethe results when we get them, if we really have uniform prior belief. Someonepoints out to us that with a Bayesian analysis we’re guaranteed to draw thewrong conclusion if θ is just under 1/2. We shrug and plan to use the Bayesiananalysis anyway.

2 Non-uniform prior belief, and a non-uniformprior distribution

What if, however, we think that θ = 1/2 is a real possibility? And when someonepoints out the Bayesian analysis would give bad results around θ = 1/2, we seethis as a real problem?

Well.... if we’re still using our prior beliefs as the prior distribution, theBayesian analysis will do fine.

For example, suppose our prior belief is that there’s a 50% chacne thatθ = 1/2, and otherwise it’s uniformly distributed. That is, our prior probability

2

density is

p(θ) =1

2δ(θ) +

1

2

I consider the posterior odds ratio:

p(θ 6= 1/2|sN )

p(θ = 1/2|sN )=p(sN |θ 6= 1/2)

p(sN |θ = 1/2)× p(θ 6= 1/2)

p(θ = 1/2)

where sN is the observed value of SN .The prior odds ratio is 1 since we decided θ = 1/2 has a prior probability of

50%, so the posterior odds ratio is the likelihood ratio.Let’s call being ”convinced” the event that the likelihood ratio is above 100,

since in this case the posterior odds that θ 6= 1/2 are 100 to 1. This is a littledifferent from what we were considering before, which us being convinced thatthe parameter is above 1/2. But the results here will still be meaningful, becauseif we can’t be convinced it’s not 1/2, we certainly can’t be convinced it’s above1/2.

The numerator of the likelihood ratio:

p(sN |θ 6=1

2) =

∫ 1

0

(N

sN

)θsN (1− θ)N−sN

The denominator of the likelihood ratio:

p(sN |θ =1

2) =

(1

2

)N

The likelihood ratio, therefore, is

2NB(SN + 1, N − SN + 1)

where B is the beta function.I did some simulations. In each ”trial”of the simulation, we sample a hundred

thousand people and calculate the log10 likelihood ratio after each person. I callthe log10 log-likelihood ratio ”ev”, the ”evidence”. When the evidence is above2, that’s when the posterior odds are 100 to 1 and we conclude that θ 6= 1/2.

As an illustration, I plot the first five trials.

> # Simulating the trials

> set.seed(552)

> # Evidence for theta is not equal to 1/2

> ev <- function(N, S) N * log10(2) + log10(exp(1)) * lbeta(S+1, N-S+1)

> trials <- data.frame(replicate(1000, ev(1:10^5, cumsum(rbinom(10^5, 1, 1/2)))))

> # Making the plot

> plot(0, xlim=c(0,10^5), ylim=c(-3,3), col='white',+ main='Evidence that theta is not 1/2',+ ylab='log10 likelihood ratio', xlab='number of trials')> for (i in 1:5) lines(trials[,i], col=rainbow(5)[i])

> abline(h=2, col='red')

3

0e+00 2e+04 4e+04 6e+04 8e+04 1e+05

−3

−2

−1

01

23

Evidence that theta is not 1/2

number of trials

log1

0 lik

elih

ood

ratio

So, while I’m not sure how to do the mathematical analysis, it looks likethey are never going to go above 2, because they decrease progressively, ratherthan fluctuating around zero.

In what fraction of my thousand such trials did the evidence cross 2?

> mean(sapply(trials, function(v) any(v>2)))

[1] 0.008

About 1% of the time. So the experimenters can only trick us into 100 to 1confidence about 1% of the time.

So, in this case, when we’re deciding whether to use a Bayesian analysis, webelieve θ might be exactly 1/2, but we also can’t be tricked, so we again thinka Bayesian analysis will end up being fine.

3 Non-uniform prior belief and uniform priordistribution

So when do we think a Bayesian analysis will fail? When is it, that we’rereviewing the protocol of the experiment, planning how to do the analysis, andwe decide that a Bayesian analysis will be manipulated?

Only when we think that θ might be around 1/2, but we plan to use auniform prior.

4

4 Is any of this meaningful?

I’ve only looked at two examples rather than deriving general theorems. Also,I’ve taken a prospective viewpoint. What about looking back on the results?Do we end up regretting deciding to use a Bayesian analysis? I haven’t reallythought that through.

But I hope I’ve clarified my original point. I’m not saying we should updateon the experimental design. Though perhaps the design should lead us to rep-resent our prior beliefs in the analysis, when ordinarily we can get away withusing priors that are higher entropy than our beliefs.

I’m just saying that the argument in the OP has no force to convince someonewho hasn’t seen the data that a Bayesian analysis will fail.

5