Optimal Testing in the Experiment-rich Regime

Sven Schmit Virag Shah Ramesh JohariStitch Fix, Inc Stanford University Stanford University

Abstract

Motivated by the widespread adoption oflarge-scale A/B testing in industry, we pro-pose a new experimentation framework forthe setting where potential experiments areabundant (i.e., many hypotheses are avail-able to test), and observations are costly; werefer to this as the experiment-rich regime.Such scenarios require the experimenter tointernalize the opportunity cost of assigninga sample to a particular experiment. We fullycharacterize the optimal policy and give analgorithm to compute it. Furthermore, wedevelop a simple heuristic that also providesintuition for the optimal policy. We use sim-ulations based on real data to compare boththe optimal algorithm and the heuristic toother natural alternative experimental designframeworks. In particular, we discuss theparadox of power: high-powered “classical”tests can lead to highly inefficient samplingin the experiment-rich regime.

1 INTRODUCTION

In modern A/B testing (e.g., for web applications), it isnot uncommon to find organizations that run hundredsor even thousands of experiments at a time (Kaufmanet al., 2017; Tang et al., 2010a; Kohavi et al., 2009; Bak-shy et al., 2014). Increased computational power andthe ubiquity of software have made it easier to generatehypotheses and deploy experiments. Organizations typ-ically continuously experiment using A/B testing. Inparticular, the space of potential experiments of inter-est (i.e., hypotheses being tested) is vast; e.g., testingthe size, shape, font, etc., of page elements, testingdifferent feature designs and user flows, testing differ-ent messages, etc. Artificial intelligence techniques are

Proceedings of the 22nd International Conference on Ar-tificial Intelligence and Statistics (AISTATS) 2019, Naha,Okinawa, Japan. PMLR: Volume 89. Copyright 2019 bythe author(s).

being deployed to help automate the design of suchtests, further increasing the pace at which new experi-ments are designed (e.g., Sensei, Adobe’s A/B testingproduct, is being used in Adobe Target).1

This abundance of potential experiments has led toan interesting phenomenon: despite the large num-bers of visitors arriving per day at most online webapplications, organizations need to constantly considerthe most efficient way to allocate these visitors to ex-periments. For many experiments, baseline rates maybe small (e.g., a low conversion rate), or more gener-ally effect sizes may be quite small even relative tolarge sample sizes. For example, large organizationsmay be seeking relative changes in a conversion rateof 0.5% or less, potentially necessitating millions ofusers allocated to a single experiment to discover atrue effect. (See Tang et al. (2010a,b); Deng et al.(2013) and Azevedo et al. (2018), where these issuesare discussed extensively.) Since organizations havea plethora of hypotheses of interest to test, there isa significant opportunity cost: they must constantlytrade off allocation of a visitor to a current experimentagainst the potential allocation of this visitor to a newexperiment.

In this paper, we study a benchmark model with thefeature that experiments are abundant relative to thearrival rate of data; we refer to this as the experiment-rich regime. A key feature of our analysis is the impactof the opportunity cost described above: whereas muchof optimal experiment design takes place in the set-ting of a single experiment, the experiment-rich regimefundamentally requires us to trade off the potentialfor discoveries across multiple experiments. Our maincontribution is a complete description of an optimaldiscovery algorithm for our setting; the developmentof an effective heuristic; and an extensive data-drivensimulation analysis of its performance against moreclassical techniques commonly applied in industrialA/B testing.

We present our model in Section 2. The formal settingwe consider mimics the setting of most industrial A/B

1https://www.adobe.com/marketing-cloud/target.html

Optimal Testing in the Experiment-rich Regime

testing contexts. The experimenter receives a stream ofobservational units and can assign them to an infinitenumber of possible experiments, or alternatives, ofvarying quality (effect size). We consider a Bayesiansetting where there is a prior over the effect size ofeach alternative, which is natural in a setting with aninfinite number of experiments.

We focus on the objective of finding an alternativethat is at least as good as a given threshold s as fastas possible. In particular, we call an alternative adiscovery if the posterior probability that the effectis greater than s is at least 1 � ↵, and the goal is tominimize the expected time per discovery. This is anatural criterion: good performance requires findingan alternative that is actually delivering practicallysignificant effects (as measured by s). Adjusting s and↵ allows the experimenter to trade off the quality andquantity of discoveries made. Note that under thiscriterion any optimal policy is naturally incentivizedto find the “best” experiments, because the discoverycriterion is easiest to be met for those alternatives.

In Section 3 we present an optimal policy for alloca-tion of observations to experiments. Since observationsarrive sequentially, the problem can be equivalentlyformulated as minimizing the cumulative number ofobservations until until a discovery is made. We char-acterize a dynamic programming approximation of thisproblem, and show this method converges to the opti-mal policy in an appropriate sense. We also develop asimple heuristic that approximates and provides insightinto the optimal policy.

In Section 4 we use data on baseball players’ battingaverages as input data for a simulation analysis ofour approach. Our simulations demonstrate that ourapproach delivers fast discovery while controlling therate of false discoveries; and that our heuristic ap-proximates the optimal policy well. We also use thesimulation setup to compare our method to “classical”techniques for discovery in experiments (e.g., hypothe-sis testing). This comparison reveals the ways in whichclassical methods can be inefficient in the experiment-rich regime. In particular, there is a paradox of power:efficient discovery can often lead to low power in a clas-sical sense, and conversely high-powered classical testscan be highly inefficient in maximizing the discoveryrate.

Due to space constraints, all proofs are given in theappendix.

1.1 Related work

The literature on sequential testing goes back manydecades. Originally, Wald and Wolfowitz (1948) pro-pose an optimal test, the sequential probability ratio

test, or SPRT for short, for testing a simple hypoth-esis. See also Chernoff (1959). For a more thoroughoverview of sequential testing, we refer the interestedreader to Siegmund (2013), Wetherill and Glazebrook(1986), Shiryayev (1978) and Lai (1997). None of theseapproaches consider the opportunity cost associatedwith having multiple experiments.

Recently, there has been an increased interest in sequen-tial testing due to the rise in popularity of A/B testing(Deng et al., 2017; Kaufman et al., 2017; Kharitonovet al., 2015; Goldberg and Johndrow, 2017), and theubiquity of peeking (Johari et al., 2017; Balsubramaniand Ramdas, 2016; Deng et al., 2016). A recent paperby Azevedo et al. (2018) discusses how the tails of theeffect distribution affect the assignment strategy of ob-servations to experiments and complements this worknicely.

There is also a strong connection to the multi-armedbandit literature (Gittins et al., 2011; Bubeck and Cesa-Bianchi, 2012), especially the pure exploration problem(Bubeck et al., 2009; Jamieson et al., 2014; Russo, 2016),where the goal is to find the best arm. The case withinfinitely many arms is studied by Carpentier and Valko(2015); Chaudhuri and Kalyanakrishnan (2017); Azizet al. (2018). Locatelli et al. (2016) studies the settingof finding the set of arms (out of finitely many) abovea given threshold in a fixed time horizon. Ramdas et al.(2017) consider a setting that combines the multi-armedbandit problems with sequential tests.

Methods to control of the false discovery rate in thesequential hypothesis setting are discussed by Fosterand Stine (2007), Javanmard and Montanari (2016)and Ramdas et al. (2017). The connection betweenwith multi-armed bandits is made by Yang et al. (2017).However, the Bayesian framework we propose does notrequire multiple testing corrections. These works takethe results of the tests as given and provide methods toadjust for multiple comparisons in a sequential manner,rather than helping the experimenter to decide whatexperiments to run.

The heavy-coin problem (Chandrasekaran and Karp,2012; Malloy et al., 2012; Jamieson et al., 2016; Laiet al., 2011) is another closely related research area.Here, a fraction of coins in a bag is considered heavy,while most are light. The goal is to find a heavy coin asquickly as possible. These approaches rely on likelihoodratios, as there are only two alternatives, and there is aconnection to the CUSUM procedure (Page, 1954). Re-cently, these ideas have been adapted to crowdfundingapplications Jain and Jamieson (2018).

Optimal stopping rules have been studied extensively,often under the umbrella of the secretary problem (Free-man, 1983; Samuels, 1991). There, the focus is on

Sven Schmit, Virag Shah, Ramesh Johari

comparing across alternatives.

2 MODEL AND OBJECTIVE

In this section we describe the model we study and theobjective of the experimenter.

Experiments. We consider a model with an infinitenumber of experiments, or alternatives, indexed byi 2 {1, 2, . . .}. Each experiment is associated with aparameter µi 2 M ⇢ R drawn independently from acommon (known) prior ⇡ that completely characterizesthe distribution of outcomes corresponding to thatexperiment. Throughout our analysis, the experimenteris interested in experiments with higher values of µi.

Actions and outcomes. At times t = 1, 2, . . ., theexperimenter selects an alternative It and observes anindependent outcome Xt drawn from a distributionF (µIt). Note, in particular, that opportunities for ob-servations arrive in a sequential, streaming fashion. Wealso assume that observations are independent acrossexperiments.

We assume that F (µi) is described by a single parame-ter natural exponential family, i.e. the density for anobservation can be written as:

fX(x | µ) = h(x) exp (µS(x)�A(µ)) , (1)

for known functions S, h, and A. Let Sit =P

t:Ik=i S(Xt) be the canonical sufficient statistic forexperiment i at time t. Note that in particular, thismodel includes the conjugate normal model with knownvariance and the beta-binomial model for binary out-comes.

Policies. Let Ft = �{X1, I1, X2, I2, . . . , Xt, It} denotethe �-field generated. A policy is a mapping from Ft

to experiments.

Discoveries. The experimenter is interested in findingdiscoveries, defined as follows.Definition 1 (Discovery). We say that alternative i

is a discovery at time t, given s and ↵, if

P(µi < s | Ft) < ↵. (2)

Here s and ↵ are parameters that capture the exper-imenter’s preferences, i.e., the level of aggressivenessand risk that she is willing to tolerate. (Note that thisis more stringent than the related false discovery rateguarantees (Benjamini and Hochberg, 2007).)

We assume that the prior satisfies P(µi < s | ;) 2 (↵, 1)to avoid the trivial scenarios that all or none of thealternatives is a discovery before trials begin.

Objective: Minimize time to discovery. As moti-vated in the introduction, informally the objective is to

find discoveries as fast as possible. We formalize thisas follows: The goal of the experimenter is to design apolicy (i.e., an algorithm to match observations to ex-periments) such that the number of observations untilthe first discovery is minimized.

In particular, define the time to first discovery ⌧ as:

⌧ = min{t : 9i⇤ s.t. P(µi⇤ < s | Ft) < ↵}. (3)

Then the goal is to minimize E[⌧ ] over all policies.Given this goal, the only decision the experimenterneeds to make at each point in time till the first suc-cess is whether to reject the current experiment or tocontinue with it.

Discussion. We conclude with three remarks regard-ing our model.

(1) Posterior validity. Note that at the (random) stop-ping time ⌧ , the posterior is computed based on thepotentially adaptive matching policy used by the ex-perimenter. The following lemma shows that whenthe experimenter computes the posterior and decidesto stop the experiment at time t when the conditionP(µi⇤ < s | Ft) is met, the decision to stop does notinvalidate the discovery.Lemma 1. The posterior for the discovered experimenti⇤ at time ⌧ satisfies

P(µi⇤ < s | F⌧ ) < ↵ (4)

almost surely.

(2) Fixed cost per experiment. In some scenarios, start-ing a new experiment has a cost; e.g., there may bea cost to implementing a new variant, or results mayneed to be analyzed on a per experiment basis. Wecan incorporate such a cost in the objective, and ourresults and approach generalize accordingly. Formally,let c be the cost of starting a new experiment, andlet mt = |{i : 9t0 t : It0 = i}| be the cumulativenumber of matched experiments up to time t. We caninclude the per experiment cost by considering insteadthe problem of minimizing E[⌧ + cm⌧ ].

(3) Fast rejection of experiments. The optimal policy,described in the next section, tends to reject manyexperiments very quickly, sometimes even after a sin-gle sample. That is not surprising, given that relatedCUSUM procedures (Page, 1954) share this characteris-tic. This happens since we are studying an asymptoticregime where the number of experiments are infinite.As we will see, the limiting case of the experiment-richregime provides a valuable lens that challenges commonperceptions, such as the importance of power.

In practice, an experimenter may wish to avoid suchan aggressive rejection since starting experiments may

Optimal Testing in the Experiment-rich Regime

be costly. Including a cost term for each experiment inour setup can help alleviate the problem, which is easyto do as described above.

3 OPTIMAL POLICY

In this section, we characterize the structure of theoptimal policy, show that it can be approximated arbi-trarily well by considering a truncated problem, andgive an algorithm to compute the optimal policy of thetruncated problem. Finally, we present a simple heuris-tic that approximates the optimal policy remarkablywell.

3.1 Sequential policies

We start with a key structural result that simplifiesthe search for an optimal policy. The following lemmashows that we can focus on policies that only considerexperiments sequentially, in the sense that once a newexperiment is being allocated observations, no previousexperiment will ever again receive observations.Lemma 2. There exists an optimal policy such thatIt+1 � It for all t almost surely.

This result hinges on three aspects of our model: exper-iments are independent of each other, with identicallydistributed effects µi; there are an infinite number ofexperiments available; and observations arrive in aninfinite stream. As a consequence, all experiments area priori equally viable, and a posteriori once the exper-imenter has determined to stop allocating observationsto an experiment, she need never consider it again.

Note in particular that this lemma also reveals thatany optimal policy for the first discovery also straight-forwardly minimizes the expected time until the k’thdiscovery, for any k.

3.2 Reformulating the optimization problem

Based on Lemma 2, we can reformulate and simplifythe optimization problem faced by the experimenter asa sequential decision problem, where the only choice iswhether or not to continue testing the current experi-ment.

We abuse notation to describe this new perspective.Let µ denote the effect size of the current experiment.In particular, let Xn be the n’th observation; let Fn

be the �-field generated by observations of the currentexperiment (X1, . . . , Xn). Let Sn =

Pnk=1 S(Xk) de-

note the canonical sufficient statistic at state n. Thestate of the sequential decision problem is (n, Sn), thenumber of observations and the sufficient statistic ofthe current experiment.

If (n, Sn) has the property that P(µ < s|Sn) < ↵, thena discovery has been found and so the process stops.The following lemma shows that this discovery criterioninduces an acceptance region on the sufficient statisticSn, i.e., a sequence of thresholds an such the currentexperiment is a discovery when Sn � an.Lemma 3. There exists a sequence {an}1n=1 such thatP(µ < s | Sn) < ↵ if and only if Sn > an.

If Sn < an, then the experimenter can make one of twodecisions:

1. Continue (i.e., collect one additional observationon the current experiment); or

2. Reject (i.e., quit the current experiment and collectthe first observation of a new experiment).

If Continue is chosen, the state updates to (n+1, Sn+1).If Reject is chosen, the state changes to (1, S1) (whereS1 is an independent draw of the sufficient statisticafter the first observation); and in either case, theprocess continues.

The goal of the experimenter is to minimize the ex-pected time until the observation process stops, i.e.,until a discovery is found. Let V (n, Sn) be this mini-mum, starting from state (n, Sn). Then the Bellmanequation for this process is as follows:

V (n, Sn) = 0, Sn � an; (5)V (n, Sn) = 1 +min {E[V (n+ 1, Sn+1)|Sn],

E[V (1, S1)]} , Sn an n � 1. (6)

The first line corresponds to the case where Sn is in theacceptance region, i.e., the process stops. In the secondline, we consider two possibilities: continuing incurs aunit cost for the current observation, plus the expectedcost from the state (n+ 1, Sn+1); rejecting resets thestate with no cost incurred. The optimal choice isfound by minimizing between these alternatives. Theexpected number of samples T ⇤ till a discovery satisfiesT

⇤ = 1 + E[V (1, S1)].

3.3 Characterizing the optimal policy

The following theorem shows that an optimal policyfor the dynamic programming problem (5)-(6) can beexpressed using a sequence of rejection thresholds onthe sufficient statistic. That is, for each n there is anrn such that it is optimal to Continue if Sn � rn, andto Reject if Sn < rn.Theorem 4. There exists an optimal policy for (5)-(6)described by a sequence of rejection thresholds {rn}1n=1

such that, after n observations, Reject is declared ifSn < rn, Continue is declared if rn Sn an, andthe process stops with a discovery if Sn > an.

Sven Schmit, Virag Shah, Ramesh Johari

The remainder of the section is devoted to computingthe optimal sequence of rejection thresholds.

3.4 Approximating the optimal policy viatruncation

In order to compute an optimal policy, we consider atruncated problem. This problem is identical in everyrespect to the problem in Section 3.2, except that weconsider only policies that must choose Reject afterk observations. We refer to this as the k-truncatedproblem.

Let Vk(n, Sn) denote the minimum expected cumu-lative time to discovery for the k-truncated prob-lem, starting from state (n, Sn). The Bellman equa-tion is nearly identical to (5)-(6), except that nowVk(k, Sk) = 1 + E[Vk(1, S1)], Sk ak, and we add theadditional constraint that n < k to (6). We have thefollowing proposition.Theorem 5. There exists an optimal policy for thek-truncated problem described by a sequence of rejec-tion thresholds {rkn}1n=1 such that, after n observations,Reject is declared if Sn < r

kn, Continue is declared if

rkn Sn an, and Accept is declared is if Sn > an.

Further, let T⇤k = E[Vk(1, S1)] + 1 be the optimal ex-

pected number of observations until a discovery is made.Then for each n, rkn ! rn as k ! 1; and T

⇤k ! T

⇤ ask ! 1.

3.5 Computing the truncated optimal policy

The truncated horizon brings us closer to computing anoptimal policy, but it is still an infinite horizon dynamicprogramming problem. In this section we show insteadthat we can compute the truncated optimal policyby iteratively solving a single-experiment truncatedproblem with a fixed rejection cost . Let Wk(n, Sn|)be the optimal expected cost for this problem startingfrom state (n, Sn). We have the following Bellmanequation.

Wk(n, Sn|) = 0, Sn > an; (7)Wk(k, Sk|) = , Sk ak; (8)Wk(n, Sn|) = 1 +min {E[Wk(n+ 1, Sn+1|)|Sn],} ,

n < k, Sn an. (9)

For any terminal cost , this dynamic programmingproblem is easily solved using backward induction tofind the rejection boundaries. The following theoremshows how we can use this solution to find an optimalpolicy to the truncated problem.Theorem 6. If = T

⇤k , then the optimal policy for

(7)-(9) with rejection thresholds rkn found by backward

induction satisfies rkn = r

kn for all n k. Furthermore,

let f() = 1+E[Wk(1, S1|)] be the optimal cost. Thenif > T

⇤k , f() < , and if < T

⇤k , then f() > .

Thus, to find approximately optimal rejection thresh-olds, select k suitably large, and start with an arbitrary. Then iteratively compute the corresponding thresh-olds r

kn and the cost f(), using bisection to converge

on T⇤k , and thus the corresponding optimal thresholds.

We note that the same program we have outlined inthis section can be used to compute an optimal policywith a per experiment fixed cost c, by using rejectioncost + c instead of . Empirically, this leads toonly slightly lower rejection thresholds; due to spaceconstraints, we omit the details.

3.6 HEURISTIC APPROXIMATION

We have seen that the optimal policy is easy to approx-imate by solving dynamic programs iteratively. How-ever, this does not give us direct insight into the struc-ture of the solution, and in certain cases a quick rule-of-thumb that provides an approximate policy might beall that is required. In this section, we show that thereexists a simple heuristic that performs remarkably well.

The approximate rejection boundary at time n is foundas follows. Let µ̂ be the MAP estimate of µ for sufficientstatistic Sn+T⇤ = an+T⇤ . Then reject the currentexperiment if Sn is not plausible under µ̂. That is, theheuristic boundary rn is, for a suitably chosen �,

P(Sn rn | µ = µ̂) = �. (10)

Of course, this heuristic is not practical as is, as ingeneral we do not know T

⇤ unless we compute theoptimal policy. But often an+t varies only little in t

so a reasonable approximate choice Th is sufficient. InFigure 1 we plot the discovery and rejection boundaries,along with the heuristic outlined above (with Th = T

⇤),for the normal and Bernoulli models.

The heuristic and optimal policies clearly exhibit aggres-sive rejection regions, cf. Figure 1. THe interpretationis as follows: to continue sampling from the currentexperiment, we do not just want its quality to be s,but substantially better than s, since an > s for all n.If not, it would take too many additional observationsto verify the discovery.

4 CASE STUDY: BASEBALL

We now empirically analyze our testing frameworkbased on a simulation with baseball data. First, wedemonstrate empirically that the proposed algorithmleads to fast discoveries, and behaves differently fromtraditional testing approaches. Second, we show that

Optimal Testing in the Experiment-rich Regime

Figure 1: Acceptance and rejection regions for the conjugate Normal and the Beta-Binomial models. The dashedblue line gives the heuristic rejection boundary, while the red line corresponds to the optimal rejection thresholds.Note that the boundaries are shown in terms of the MAP estimate.

the rule-of-thumb heuristic performance is close to thatof the optimal policy.2

Data We use the baseball dataset with pitching andhitting statistics from 1871 through 2016 from theLahman R package. The number of At Bats (AB)and Hits (H) is collected for each player, and we areinterested in finding players with a high batting average,defined as bi = Hitsi/At Batsi. We consider playerswith at least 200 At Bats, which leaves a total of5721 players, with a mean of about 2300 At Bats.In the top left of Figure 2, we plot the histogram ofbatting averages, along with an approximation by abeta distribution (fit via method of moments). We notethat these fit the data reasonably well, but not perfectly.This discrepancy helps us evaluate the robustness to amisspecified prior.

Simulation setup To construct the testing problem,we view the batters as alternatives, with empiricalbatting average bi of batter i treated as ground truth.We want to find alternatives with bi > s. We draw aBernoulli sample of mean bi to simulate an observationfrom alternative i. These samples are then used to testwhether bi > s. We set ↵ = 0.05, and vary s between0.25 and 0.32. For each simulation, we iterate througheach batter and repeat it 1000 times to reduce variance.This allows us to compare methods fairly, ensuring thateach procedure is run on exactly the same test cases.

4.1 Benchmarks

To assess performance, we compare several testing pro-cedures. Note that the non-traditional setup of our test-ing framework does not allow for easy comparison withother methods, in particular frequentist approaches,as they give different guarantees. Thus, we restrict

2Code to replicate results will be made publicly available.

attention to Bayesian methods that provide the sameerror guarantee. All of the benchmarks use the samebeta prior computed above.

Optimal policy First we study the optimal policybased on the beta-binomial model, computed usingthe bisection and backward induction approach in Sec-tion 3.5, where we truncate after k = 5000 samples.

Heuristic policy Next, we include the heuristic re-jection thresholds that approximate the optimal policyfor truncation k = 5000 samples. The heuristic policyrequires setting two parameters: Th, i.e., how far tolook into the future to find the acceptance boundary,which is ideally set close to T

⇤; and the rejection region�. To demonstrate the the insensitivity to T

⇤, we useTh = 2000 and � = 0.2 for all simulations. (Note thatT

⇤ varies dramatically as we change the threshold s.

Fixed sample size test Our next benchmark is asimple fixed sample size test. For each experiment,we gather N observations, and claim a discovery ifP (µi < s | Yi) < ↵ where Yi is the number of Hits ofalternative (batter) i. We focus our attention on usingN = 1000 samples per test, as this seems to performbest when compared to other sample sizes, but anydifferences are immaterial for our conclusions.

Fixed sample size test with early stopping Thisbenchmark is similar to the fixed sample size test, ex-cept that we stop the experiment early if the discoverycriterion is met. Thus, we can quantify the gains frombeing able to discover early.

Bayesian sequential test Now we consider a se-quential test that also rejects early. In particular, wereject the current experiment if P(bi > s | Si

t) < �. Wealso reject an alternative after 4000 samples. This ap-

Sven Schmit, Virag Shah, Ramesh Johari

proach also requires careful tuning of �. In particular,if � is too large, say larger than the prior probabilityP0(bi > s), then the test is too aggressive and rejectsall alternatives outright. Instead, we found empiri-cally that setting � = 0.9P0(bi > s) leads to goodperformance across all values of s.

4.2 Results

Average time to discovery The average numberof observations until a discovery is shown in the topright plot of Figure 2. As expected, the fixed sampletest performs worst. Early stopping leads to slightlybetter performance, but this method is still not effec-tive as most of the gains come from early rejection.The Bayesian sequential test demonstrates this effectand shows substantial gains over the fixed tests. Theheuristic policy, despite lack of parameter tuning, per-forms very well, essentially matching the performanceof the optimal algorithm for most thresholds.

False discovery proportion and robustnessNext, we compare the false discovery proportion (FDP)(Benjamini and Hochberg, 2007), i.e., the fraction ofdiscoveries that in fact had true bi < s. If the prioris correctly specified, the methods we consider satisfyE(FDP) ↵.3 Indeed, we observe that the guaran-tee holds for most thresholds and algorithms in thebottom left plot of Figure 2. There is some minorexceedance of the FDP for thresholds around s ⇡ 0.28,which can be explained by the fact that the prior doesnot fit the empirical batting averages perfectly. Sincethere are few rejections for thresholds beyond s = 0.3,the FDP estimate has higher variance in that regime.Across all simulations, the optimal policy has an FDPof 0.048 < ↵. Finally, we see that the lack of earlystopping makes the fixed test rather conservative.

The paradox of power Finally, we compare power,i.e., the fraction of alternatives i with bi > s that aredeclared a discovery. Power comparisons across thealgorithms are plotted in the bottom right of Figure 2.The most surprising insight from the simulations isthe paradox of power. Algorithms that are effectivehave very low power. This is counter-intuitive: howcan algorithms that make many discoveries have onlya small chance of picking up true effects? The maindriver of good performance for an algorithm is the abil-ity to quickly reject unpromising alternatives. Someunpromising alternatives are “barely winners”: i.e., biis only slightly above s. In the experiment-rich regime,such alternatives should be rejected quickly, becauseit takes too many observations to get enough concen-

3However, note that this is different from frequentistFDR guarantees, which these methods do not provide.

tration around the posterior to claim a discovery. Thiseffect leads to low power, but fast discoveries.

5 CONCLUSION

We consider an experimentation setting where observa-tions are costly, and there is an abundance of possibleexperiments to run — an increasingly prevalent scenarioas the world is becoming more data-driven. Based onbackward induction, we can compute an approximatelyoptimal algorithm that allocates observations to exper-iments such that the time to a discovery is minimized.Simulations validate the efficacy of our approach, andalso reveal discuss the paradox of power : there is atension between high-powered tests, and being efficientwith observations.

Our paradigm has several additional practical benefits.First, we can leverage knowledge across experimentsthrough the prior. Second, adaptive matching of obser-vations to experiments does not preclude valid inference,and thus outcomes can thus continuously be monitored.Finally, the framework also provides an easy “user in-terface”: it directly incorporates the desired effect size,and leads to guarantees that are easy to explain tonon-experts.

ACKNOWLEDGEMENTS

We would like to thank Johan Ugander, David Walsh,Andrea Locatelli, and Carlos Riquelme for their helpfulcomments and suggestions. This research was donewhile Sven Schmit was at Stanford University. Theresearch was supported by the National Science Foun-dation under grants 1544548 and 1839229.

References

E. M. Azevedo, A. Deng, J. M. Olea, J. M. Rao, andE. G. Weyl. A/b testing. In Proceedings of the Nine-teenth ACM Conference on Economics and Compu-tation. ACM, 2018.

M. Aziz, J. Anderton, E. Kaufmann, and J. A. Aslam.Pure exploration in infinitely-armed bandit modelswith fixed-confidence. In ALT, 2018.

E. Bakshy, D. Eckles, and M. S. Bernstein. Designingand deploying online field experiments. In Proceed-ings of the 23rd international conference on Worldwide web, pages 283–292. ACM, 2014.

A. Balsubramani and A. Ramdas. Sequential nonpara-metric testing with the law of the iterated logarithm.CoRR, abs/1506.03486, 2016.

A. Y. Benjamini and Y. Hochberg. Controlling the falsediscovery rate: A practical and powerful approachto multiple testing. 2007.

Optimal Testing in the Experiment-rich Regime

0.15 0.20 0.25 0.30 0.35batting avHragH

0.0

2.5

5.0

7.5

10.0

12.5

15.0

frHq

uHnF

y

HistograP H/AB

BHta(56.9, 167.9)data

0.25 0.26 0.27 0.28 0.29 0.30 0.31 0.32thrHshold

103

104

105

#sa

Ppl

Hs /

#di

sFov

HriH

s

AvHragH tiPH to disFovHry

0.25 0.26 0.27 0.28 0.29 0.30 0.31 0.32thrHshold

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

#fa

lsH

disF

ovHr

iHs

/ #di

sFov

HriH

s

)alsH disFovHry proportion

0.25 0.26 0.27 0.28 0.29 0.30 0.31 0.32thrHshold

0.0

0.1

0.2

0.3

0.4

0.5

0.6

#tr

uH d

isFo

vHriH

s / #

good

bat

tHrs

3owHr

optiPalhHuristiFfixHdfixHd, Harly stoppingbayHsian

Figure 2: Top left: histogram of batting averages. Top right: Efficacy of of algorithms. Bottom left: Plot thefalse discovery proportion across thresholds. Bottom right: Plot of the empirical power of algorithms. Note theparadox of power effect: the most efficient algorithms have low power.

S. Bubeck and N. Cesa-Bianchi. Regret analysis ofstochastic and nonstochastic multi-armed banditproblems. Foundations and Trends in MachineLearning, 5:1–122, 2012.

S. Bubeck, R. Munos, and G. Stoltz. Pure explorationin multi-armed bandits problems. In ALT, 2009.

A. Carpentier and M. Valko. Simple regret for infinitelymany armed bandits. In ICML, 2015.

K. Chandrasekaran and R. M. Karp. Finding the mostbiased coin with fewest flips. CoRR, abs/1202.3639,2012.

A. R. Chaudhuri and S. Kalyanakrishnan. Pac identifi-cation of a bandit arm relative to a reward quantile.In AAAI, pages 1777–1783, 2017.

H. Chernoff. Sequential design of experiments. TheAnnals of Mathematical Statistics, 30(3):755–770,1959.

A. Deng, Y. Xu, R. Kohavi, and T. Walker. Improvingthe sensitivity of online controlled experiments byutilizing pre-experiment data. In Proceedings of thesixth ACM international conference on Web searchand data mining, pages 123–132. ACM, 2013.

A. Deng, J. Lu, and S. Chen. Continuous monitor-ing of a/b tests without pain: Optional stoppingin bayesian testing. 2016 IEEE International Con-

ference on Data Science and Advanced Analytics(DSAA), pages 243–252, 2016.

A. Deng, J. Lu, and J. Litz. Trustworthy analysis ofonline a/b tests: Pitfalls, challenges and solutions.In WSDM, 2017.

D. P. Foster and R. A. Stine. Alpha-investing: Aprocedure for sequential control of expected falsediscoveries. 2007.

P. Freeman. The secretary problem and its extensions:A review. International Statistical Review/RevueInternationale de Statistique, pages 189–206, 1983.

J. Gittins, K. Glazebrook, and R. Weber. Multi-armedbandit allocation indices. John Wiley & Sons, 2011.

D. Goldberg and J. E. Johndrow. A decision the-oretic approach to a/b testing. arXiv preprintarXiv:1710.03410, 2017.

L. Jain and K. Jamieson. Firing bandits: Optimiz-ing crowdfunding. In International Conference onMachine Learning, pages 2211–2219, 2018.

K. G. Jamieson, M. Malloy, R. D. Nowak, andS. Bubeck. lil’ ucb : An optimal exploration al-gorithm for multi-armed bandits. In COLT, 2014.

K. G. Jamieson, D. Haas, and B. Recht. The power ofadaptivity in identifying statistical alternatives. InNIPS, 2016.