A Nonparametric Sequential Test for Online RandomizedExperiments

Vineet AbhishekWalmart Labs

Sunnyvale, CA, USAvabhishek@walmartlabs.com

Shie MannorTechnion

Haifa, Israelshie@ee.technion.ac.il

ABSTRACTWe propose a nonparametric sequential test that aims to ad-dress two practical problems pertinent to online randomizedexperiments: (i) how to do a hypothesis test for complexmetrics; (ii) how to prevent type 1 error inflation undercontinuous monitoring. The proposed test does not requireknowledge of the underlying probability distribution generat-ing the data. We use the bootstrap to estimate the likelihoodfor blocks of data followed by mixture sequential probabilityratio test. We validate this procedure on data from a majoronline e-commerce website. We show that the proposed testcontrols type 1 error at any time, has good power, is robust tomisspecification in the distribution generating the data, andallows quick inference in online randomized experiments.

KeywordsSequential tests; the bootstrap; A/B tests

1. INTRODUCTIONMost major online websites continuously run several ran-

domized experiments to measure the performance of newfeatures or algorithms and to determine their successes orfailures; see [10] for a comprehensive survey. A typical work-flow consists of: (i) defining a success metric; (ii) performinga statistical hypothesis test to determine whether or not theobserved change in the success metric is significant. Both ofthese steps, however, are nontrivial in practice.

Consider a typical product search performance data col-lected by any online e-commerce website. Such data is mul-tidimensional, containing dimensions such as the number ofsearch queries, the number of conversions, time to the firstclick, total revenue from the sale, etc., for each user session.The success metric of interest could be a complicated scalarfunction of this multidimensional data rather than some sim-ple average; e.g., a metric of interest could be the ratio ofexpected number of successful queries per user session to theexpected number of total queries per user session, an empiri-cal estimate of which is a ratio of two sums. Furthermore,

c©2017 International World Wide Web Conference Committee (IW3C2),published under Creative Commons CC BY 4.0 License.WWW’17 Companion, April 3–7, 2017, Perth, Australia.ACM 978-1-4503-4914-7/17/04.http://dx.doi.org/10.1145/3041021.3054196

.

some commonly used families of probability distributionssuch as the Bernoulli distribution or the normal distributionare almost never suitable to model this data. The searchqueries in a user session are highly correlated; using theBernoulli distribution to model even simple metrics such theclick through rate per impression is inaccurate.1 Anotherexample is the revenue based metrics with a considerablemass at zero and a heavy tail distribution for the nonzerovalues. The use of the normal distribution for modeling rev-enue per user session is inaccurate and sensitive to extremevalues unless the sample size is huge. As a result, the useof the binomial proportion test or the t-test for hypothesistesting is unsuitable for such metrics respectively. It requiresconsiderable effort to find appropriate models for each ofthese metrics and it is implausible to come up with a suitableparametrized family of probability distributions that couldbe used to model different complex success metrics.

Additionally, classical statistical tests such as the t-testassume that the number of samples is given apriori. Thisassumption easily breaks in an online setting where an enduser with not much formal training in statistics has nearreal-time access to the performance of ongoing randomizedexperiments: the decision to stop or continue collecting moredata is often based on whether or not the statistical teston the data collected so far shows significance or not. Thispractice of chasing significance leads to highly inflated type 1error.2 Several recent articles have highlighted this aspectof data collection as one of the factors contributing to thestatistical crisis in science; see, e.g., [18] and [5]. Use ofsample size calculator to determine the number of samples tobe collected and waiting until the required number of sampleshave been collected is hard to enforce in practice, especiallyin do-it-yourself type platforms commonly used to set up andmonitor online randomized experiments. Additionally, thesample size calculation requires an estimate of the minimumeffect size and the variance of the metric. These estimatesare often crude and result in waiting longer than necessaryin case the true change is much larger than the estimate.

The above two challenges point to two separate fields ofstatistics: the bootstrap and sequential tests. The bootstrapis frequently used to estimate the properties of complicated

1A similar observation is made by [1]. The authors suggestavoiding the use of standard formulas for standard deviationcomputation which assume independence of the samples andrecommend the use of the bootstrap.2The inflation of type 1 error because of data collection toreach significance can be explained by the law of the iteratedlogarithm, as observed in [14]. Section 5 provides some plotsto demonstrate the ill effects of chasing significance.

610

statistics; see [4] and [7] for a comprehensive review. Sequen-tial tests provide an alternative to committing to a fixedsample size by providing type 1 error control at any timeunder continuous monitoring; see [12] for a survey on sequen-tial tests. Sequential tests, however, are still uncommon inpractice partly because of the information they require aboutthe data generation process.

Our main contribution is in bringing together the ideasfrom the bootstrap and sequential tests and applying themto construct a nonparametric sequential test for performinga hypothesis test on complex metrics. We provide empiricalevidence using the data from a major online e-commercewebsite demonstrating that our nonparametric sequentialtest is suitable for the diverse type of metrics, ensures thattype 1 error is controlled at any time under continuous mon-itoring, has good power, is robust to misspecification in thedistribution generating the data, and enables making quickinference in online randomized experiments. We advocatethe use of this general approach in the absence of knowledgeof the right distribution generating the data, which typicallyis the case.

Briefly, we compute a studentized statistic for the metric ofinterest on blocks of data and use the bootstrap to estimate itslikelihood in a nonparametric manner. The likelihood ratioscan then be used to perform mixture sequential probabilityratio test (henceforth, mixture SPRT) [17]. We highlighttheoretical results from existing literature justifying the useof the bootstrap based likelihood estimation and the mixtureSPRT.

Related work: The bootstrap based nonparametric ap-proach we take distinguishes our work from the related liter-ature on sequential testing. Most of the existing literatureassumes the knowledge of parametrized family of distributionfrom which data (typically scalar) is generated; ours doesnot. The idea of sequential probability ratio test (SPRT) fortesting a simple hypothesis and its optimality originates fromthe seminal work of [20]. Mixture SPRT [17], described inSection 3.2, extends this to a composite hypothesis. For theexponential family of distributions, mixture SPRT is shownto be a test with power 1 and almost optimal with respectto expected time to stop; see [17] and [16]. [8] formalizes thenotion of always valid p-value and uses mixture SPRT forthe sequential test. MaxSPRT [11], [9] is another popularsequential test where the unknown parameter is replacedby its maximum likelihood estimate computed from the ob-served data so far and the rejection threshold is determined(typically using numerical simulations) to ensure type 1 errorcontrol. The computation of the maximum likelihood esti-mate as well as the rejection threshold requires knowing theparametrized family of distribution; consequently, MaxSPRThas been studied only for a few simple distributions. Groupsequential tests [15] divide observations into blocks of datafor inference. It, however, assumes the normal distributionand uses mean as the metric of interest. The theoretical foun-dations of the bootstrap likelihood estimation can be foundin [6]; it shows that the bootstrap based likelihood estimationof the studentized statistic provide a fast rate of convergenceto the true likelihood. [2] uses a nested bootstrap to computethe likelihood which can be computationally expensive. [3]uses confidence intervals to compute the likelihood and showsthat for the exponential family of distributions, the confi-dence interval based approach has the rate of convergence ofthe same order as in [6].

2. MODEL AND ASSUMPTIONSLet X ∈ Rd be a random vector with the CDF F (x; θ, η)

and corresponding pdf f(x; θ, η). Here θ is the scalar param-eter of interest and η is a nuisance parameter (possibly nullor multidimensional). Denote a realized value of the random

vector X by x. For j < k, let Xj:k , (Xj ,Xj+1, . . . ,Xk)be a collection of i.i.d. random vectors indexed from j to k;define xj:k similarly.

We are interested in testing the null hypothesis that θ = θ0against the alternate hypothesis θ 6= θ0 such that type 1error is controlled at any given time and the power of thetest should be high; i.e., we are looking for a function Dθ0

mapping sample paths x1,x2, . . . to [0, 1] such that for anyα ∈ (0, 1):

Pθ [Dθ0(X1:n) ≤ α for some n ≥ 1]

{≤ α if θ = θ0,≈ 1 if θ 6= θ0,

(1)where X’s are assumed to be realized i.i.d. from F (x; θ, η)and Pθ is the probability under F (x; θ, η). The first partof (1) is a statement on type 1 error control: if the nullhypothesis is true and our tolerance for incorrect rejectionof the null hypothesis is α, then the probability that the teststatistic exceeds α and rejects the null hypothesis (type 1error) is at most α. The second part of (1) is a statementon the power of the test: if the null hypothesis is false thenthe probability that the test statistic exceeds α and rejectsthe null hypothesis (the power of the test) is close to one.

Let F̂ (x1:n) be the empirical CDF constructed from ni.i.d. samples. We make the following assumption about theparameter of interest θ:

Assumption 1. Assume that θ is a functional statistic withasymptotically normal distribution, i.e., θ = T (F ) for some

functional T and√n(T (F̂ (X1:n))− θ)→ N (0, ζ2) in distri-

bution as n→∞, where T (F̂ (x1:n)) is the plug-in estimateof θ. Furthermore, assume that the asymptotic variance ζ2

does not depend on the specific choice of θ.

Asymptotic normality of plug-in statistic in Assumption 1holds under fairly general conditions and captures severalstatistics of interest; see [13] for a formal discussion on this.

3. PRELIMINARIESOur work leverages ideas from the bootstrap and mixture

SPRT. This section provides their brief overview.

3.1 The bootstrapLet Z1:n be n i.i.d. random variables and let Hn(z) ,

P [T (Z1:n) ≤ z] be the sampling distribution of a functional T .The CDF of Z’s is usually unknown and hence Hn cannot becomputed directly. The (nonparametric) bootstrap is used toapproximate the sampling distribution Hn using a resamplingtechnique. Let Z∗1:n be n i.i.d. random variables each with theCDF equal to the empirical CDF constructed from Z1:n. LetH∗n(z) , P [T (Z∗1:n) ≤ z | Z1:n] be the bootstrap distributionof the functional T . Notice that H∗n is a random CDF thatdepends on the realized values of Z1:n. The idea is to useH∗n as an approximation for Hn.3

3Under some regularity conditions and a suitable distancemetric on the space of CDFs, distance between Hn and H∗ngoes to zero with probability 1 as n→∞; see [7].

611

The bootstrap distribution is computed using Monte Carlomethod by drawing a large number of samples, each of size n,from the empirical distribution. This is identical to samplingwith replacement and provides a convenient computationalmethod to deal with complex statistics. Even in the caseswhere the statistics are asymptotically normal, we might haveto rely on the bootstrap to compute its asymptotic variance.Moreover, for asymptotically normal statistics, the bootstraptypically provides an order improvement in convergence rateto the true distribution (see [7] and [6]). This motivates theuse of the bootstrap in our nonparametric sequential test.

3.2 Mixture SPRTIn a general form, mixture SRPT works as follows. Let

ρn(z1:n; θ) be the joint pdf parametrized by θ of a sequenceof random variables Z1:n. Let θ ∈ I and π(θ) be any prioron θ with π(θ) > 0 for θ ∈ I. Define:

L(z1:n; θ0) ,

∫θ∈I ρn(z1:n; θ)π(θ)

ρn(z1:n; θ0). (2)

If θ = θ0, the sequence L(Z1:n; θ0)’s forms a martingale withrespect to the filtration induced by the sequence Zn’s.4 UsingDoob’s martingale inequality,

Pθ0[L(Z1:n; θ0) >

1

αfor some n ≥ 1

]≤ α for any α > 0,

(3)where the probability is computed assuming θ = θ0. Animmediate consequence of this is control of type 1 error atany time. The optimality of mixture SPRT with respect ofpower and the expected time to stop ([17] and [16]) and itsease of implementation motivates its use in our nonparametricsequential test.

The p-value at time n is defined as:

pvalue(z1:n; θ0) , min

{1,

1

max{L(z1:t; θ0) : t ≤ n}

}. (4)

It follows immediate from (3) and (4) that

Pθ0 [pvalue(Z1:n; θ0) ≤ α] ≤ α for any n (5)

and pvalue(z1:n; θ0) is nonincreasing in n. In this sense, this isan always valid p-value sequence, defined in [8]. The dualitybetween the p-value and the confidence interval can be usedto construct an always valid (though possibly larger than theexact) confidence interval.

Mixture SPRT, however, requires the knowledge of thefunctional form of the pdf up to the unknown parameter ofinterest. This is hard especially in a multidimensional settingand in the presence of nuisance parameter(s). We addressthis using the bootstrap, as described next.

4. THE NONPARAMETRIC SEQUENTIALTEST

This section shows how mixture SPRT and the bootstrapcould be used together to construct a nonparametric sequen-tial test for complex statistics. We will also discuss how toapply the ideas developed here to A/B tests.

4This follows from a general observation that if Z1:n ∼ρn(z1:n) and ψn(z1:n) is any other probability density func-tion, then Ln = ψn(z1:n)/ρn(z1:n) is a martingale.

4.1 The bootstrap mixture SPRTOur main idea is as follows: (i) divide the data into blocks

of size N ; (ii) compute a studentized plug-in statistic forθ on each block of data; (iii) estimate the likelihood ofthe studentized plug-in statistic for each block using thebootstrap; (iv) use mixture SPRT by treating each block asa unit observation. We provide details below.

Let x(k) , xkN :(k+1)N−1 denote the kth block of data ofsize N and s(x(k); θ) be the studentized plug-in statistic

computed from the kth data block, i.e.,

s(x(k); θ) =T (F̂ (x(k)))− θσ(T (F̂ (x(k))))

, (6)

where σ(T (F̂ (x(k))) is a consistent estimate of the standard

deviation of the plug-in statistic T (F̂ (X(k))).5 For notational

convenience, we use θ̂(k) and sθ(k) in lieu of T (F̂ (x(k))) ands(x(k); θ) respectively, with the dependence on underlyingx’s and N being implicit; and their uppercase versions todenote the corresponding random variables. The randomvariable Sθ(k) is asymptotically distributed as N (0, 1) at rateO(1/

√N) assuming θ is the true parameter. Hence for large

N , the distribution of Sθ(k) is approximately pivotal (i.e.,the distribution does not depend on the unknown θ and theeffect of η is negligible). This suggests using the normaldensity as the pdf for Sθ(k) and carry out mixture SPRTfollowing the steps in Section 3.2. The density, however, canbe approximated better using the bootstrap.

Algorithm 1 below describes the bootstrap mixture SPRT.Correctness of Algorithm 1: We give a heuristic ex-

planation of the correctness of Algorithm 1; a rigorous proofof type 1 error control or optimality of Algorithm 1 in somesense is extremely hard.6 Ln in step 3 is similar to (2).If Ln is approximately a martingale for θ = θ0, the argu-ments of Section 3.2 would imply type 1 error control at anytime under continuous monitoring. Step 2 of Algorithm 1is approximating the true pdf of the random variable Sθ(k),

denoted by gθ(k), using the bootstrap based estimate g∗(k).We only need to ensure that g∗(k) is a good approximation

of gθ0(k), assuming θ = θ0, for the martingale property to

hold. A subtlety arises because the bootstrap distribution isdiscrete. This, however, is not a problem: [6] shows that asmoothened version of bootstrap density obtained by addinga small independent noise N (0, 1/Nc) with arbitrary largec to the bootstrap samples converges uniformly to the truedensity at rate Op(1/N). Working with the studentized statis-tic is crucial in obtaining Op(1/N) convergence; see [7] forfurther details. By contrast, approximating g(k) by the nor-mal density is accurate to O(1/

√N). The bootstrap samples

behave as if realized from a true density. A Gaussian kerneldensity can be used to estimate the bootstrap density froma finite number of bootstrap samples, as in step 2 of Algo-rithm 1. The independence of the asymptotic variance andθ by Assumption 1 allows us to compute the likelihood for

5Formally,√nσ(T (F̂ (X1:n))) → ζ with probability 1 as

n→∞, where ζ is defined in Assumption 1.6The results on the optimality of mixture SPRT are knownonly for exponential families of distributions. We, however,take a nonparametric approach. A theory of bootstrap isbased on Edgeworth expansion [7]. Applying Edgeworthexpansion to analyze likelihood ratio statistic defined in (8)is quite challenging.

612

Algorithm 1 The bootstrap mixture SPRT

Given blocks of observations x(k) for k = 1, 2, . . ., and a priorπ(θ):

1. Draw θ̃1, θ̃2, . . . θ̃M i.i.d. from π(θ) for some large M .

2. For each k:

(a) Compute θ̂(k) and its standard deviation σ(θ̂(k)).

(b) Draw a random sample of size N with replacementfrom x(k); denote it by x∗(k). Repeat this B times

to obtain a sequence x∗b(k), 1 ≤ b ≤ B. For each b,

compute the studentized plug-in statistic s∗b(k) as:

s∗b(k) =T (F̂ (x∗b(k)))− θ̂(k)σ(T (F̂ (x∗b(k))))

. (7)

Let g∗(k)(·) be the pdf obtained using a Gaussian

kernel density estimate from the sequence s∗b(k),1 ≤ b ≤ B.

3. Compute Ln as follows:

Ln =

M∑m=1

∏nk=1 g

∗(k)

(θ̂(k)−θ̃mσ(θ̂(k))

)∏nk=1 g

∗(k)

(θ̂(k)−θ0σ(θ̂(k))

) . (8)

4. Reject the null hypothesis θ = θ0 if Ln > 1/α for somen > 1.

different values of θ. Step 4 is a Monte Carlo computationof Bayesian average in the right side of (2).

4.2 Application to A/B testsIn A/B tests, the goal is to identify whether or not the

success metric computed on each of the two groups of datahas a nonzero difference (i.e., θ0 = 0). The bootstrap mixtureSPRT easily extends to A/B tests. Assume for simplicitythat the block size is the same for both groups. Let x(k)’s bedata from group A (control) and y(k)’s the data from group

B (variation). We modify the quantities θ̂(k), σ(θ(k)), and

s∗b(k) of Algorithm 1 as follows:

θ̂(k) = T (F̂ (y(k)))− T (F̂ (x(k))),

σ(θ̂(k)) =

√σ2(T (F̂ (y(k)))) + σ2(T (F̂ (x(k)))),

s∗b(k) =T (F̂ (y∗b(k)))− T (F̂ (x∗b(k)))− θ̂(k)√σ2(T (F̂ (x∗b(k)))) + σ2(T (F̂ (x∗b(k))))

.

We conclude with a discussion on how to determine variousparameters in Algorithm 1. Type 1 error control holds forany prior π(θ) with nonzero density on possible values ofθ. The choice of prior, however, can affect the number ofobservations needed to reject the null hypothesis. A simplechoice is N (0, τ2); τ can be determined from the outcomesof the past A/B tests or can be set to a few percent of thereference value, computed from the data collected beforethe start of the A/B test, of the metric of interest. Thestandard deviation in (7) for each b can be computed usingthe bootstrap again on x∗b(k). This, however, is computa-

tionally prohibitive. In practice, the standard deviation iscomputed using the delta method or the jackknife; see e.g.,[4]. A similar method can be used to compute the standarddeviation in (6). The parameters B and M are merely usedfor Monte Carlo integration; any sufficiently large value isgood. Typical choices for the number of bootstrap sample Bis 1000− 5000 and the number of Monte Carlo samples M is5000− 10, 000. Because we work with a block of data at atime, the computational overhead is not much; computationcan be performed in an online manner as new blocks of dataarrive or using a MapReduce approach on the historical data.Finally, a small block size is intuitively expected to enablefast inference. Doob’s martingale inequality is far from be-ing tight for martingales stopped at some maximum lengthand becomes tighter as this maximum length is increased.However, the small block size may reduce the accuracy ofthe bootstrap likelihood estimate. Additionally, data updatefrequency of a near real-time system puts a lower limit onhow small the block of data can be; it is not uncommon toreceive a few hundred or a few thousand data points perupdate. Section 5 describes how to estimate a reasonableblock size using post A/A tests.

5. EMPIRICAL EVALUATION AND DISCUS-SION

Datasets, metrics, and baselines: We use data ob-tained from a major online e-commerce website on productsearch performance. Each dataset contains a tiny fraction ofrandomly chosen anonymized users; and for each user session,it contains the session timestamp, the number of distinctsearch queries, the number of successful search queries, andthe total revenue. Datasets contain about 1.8 million usersessions each and span nonoverlapping 2-3 weeks timeframe.We show here the results for only one dataset; similar resultswere observed for the other datasets.

We work with two metrics: (i) query success rate per usersession; (ii) revenue per user session. The former is the ratioof the sum of successful queries per user session to the sumof total queries per user session, where the sum is over usersessions. The latter is a simple average, however, with mostvalues being zero and nonzero values showing a heavy tail.These two very different metrics are chosen to evaluate theflexibility of the bootstrap mixture SPRT. We have usedN (0, τ2) distribution as prior π(θ), with τ being equal to afew percent (less than 5%) of the values of metrics computedon data before the start timestamp of the dataset underconsideration. Data is partitioned into blocks based on theorder induced by the user session timestamp.

Existing sequential tests require knowing a parameterizeddistribution generating the data. The data generation modelfor the two metrics we study in our experiments is hard tofind. We are not aware of any existing sequential tests thatcan be applied to the two complex metrics we use in ourexperiments. Consequently, we use the z-test on the entiredata as a baseline for comparison.7 The use of z-test (or thet-test) is pervasive in commercial A/B tests platforms, oftenwithout much scrutiny. Additionally, we compare the boot-strap mixture SPRT with MaxSPRT ([11], [9]) on syntheticdata generated from the Bernoulli distribution. MaxSPRTrequires knowing a parametrized likelihood function and has

7Because of the large size of the dataset we use, there is noobservable difference between the t-test and the z-test.

613

been applied mostly when the data is generated from sim-ple distributions such as the Poisson distribution and theBernoulli distribution.

Post A/A tests based methodology: We make use ofpost A/A tests to estimate the type 1 error. For each dataset,we randomly divide the users into two groups, perform thestatistical test for the null hypothesis of zero difference inthe metrics for the two groups, and compute the p-values.This process is repeated multiple times (250 in our exper-iments). Because there is no true difference between thetwo groups, the p-value should theoretically follow uniformdistribution between 0 and 1. Consequently, the Q-Q plotsfor the realized p-values from different random splits andthe uniform distribution should be the line of slope 1. Toestimate the power of the test, for each random split of thedataset, we add to the realized difference in the metrics forthe two groups some small positive offset. For each choiceof the additive offset, we then check how many times thenull hypothesis is rejected using p-value ≤ 0.05 as our nullhypothesis rejection criteria.

Type 1 error: Figure 1a shows a huge inflation of type 1error under sequential monitoring in case of classical fixedsample size z-test. The Q-Q plot lies well below the lineof slope 1. Here, the p-values are computed daily usingthe data collected until that day. The Q-Q plot is for theminimum p-value observed from daily observations, modelingthe scenario where one waits until the data shows significance.Waiting until the p-value is ≤ 0.05 and stopping the test onthat day resulted in type 1 error rate of at least 30%. Bycontrast, the Q-Q plot of always valid p-value in the case ofthe bootstrap mixture SPRT with the block size 5000 is closeto the line of slope 1, as shown in Figure 1b. This impliesthat the bootstrap mixture SPRT controls type 1 error atany time under continuous monitoring. The p-value is thefinal one at the end of all the blocks (recall that the p-valueis nonincreasing in this case).

The choice of the block size of 5000 is determined bypost A/A tests with different block sizes, looking at thecorresponding Q-Q plots, and making sure it stays abovebut close to the line of slope 1. Smaller block size makes thebootstrap approximation inaccurate while the larger blocksize makes this procedure more conservative in case of someupper limit on the number of samples available for the test.This is a consequence of fewer intermediate observationsin a finite length dataset, resulting in Doob’s martingaleinequality far from being tight. Figure 2a and 2b show theQ-Q plot for block length of 10, 000 and 20, 000 respectively.The plots lie above the line of slope 1, showing a moreconservative type 1 error control.

Test power and average duration: Next, we comparethe power of the bootstrap mixture SPRT with the z-test onthe entire data. We use the delta method to compute thestandard deviation for the query success rate; see [19]. Weuse an additive offset to the realized difference, as describedearlier in this section. Figure 3a shows the power of the testas a function of the true difference percentage for the case ofquery success rate. Here the true difference percentage is theadditive offset normalized by the value of the metric on theentire dataset. Notice some loss in power compared to thez-test on the entire dataset. This loss in power is expectedbecause of the sequential nature of the procedure on a finitedata set; equivalently, the bootstrap mixture SPRT requiresa larger number of samples to achieve the same power as the

0.00.2 0.4 0.6

0.8 1.0Theoretical Quantiles0.0

0.2

0.4

0.6

0.8

1.0

Sam

ple

Qua

ntile

s

query success rate per user session

0.00.2 0.4 0.6

0.8 1.0Theoretical Quantiles

Sam

ple

Qua

ntile

s

revenue per user session

(a) p-values under sequential observation for the z-test.

0.00.2 0.4 0.6

0.8 1.0Theoretical Quantiles0.0

0.2

0.4

0.6

0.8

1.0

Sam

ple

Qua

ntile

s

query success rate per user session

0.00.2 0.4 0.6

0.8 1.0Theoretical Quantiles

Sam

ple

Qua

ntile

s

revenue per user session

(b) p-values under the bootstrap mixture SPRT.

Figure 1: Q-Q plot of p-values computed on random splitsof the dataset.

fixed sample size tests. This is the price paid for the flexibilityof continuous monitoring. The loss in power quickly becomesnegligible as the true difference becomes large. Focusing onthe revenue metric, we notice something surprising: the z-test on the entire data has smaller power than the bootstrapmixture SPRT, as shown in Figure 3b. In this case, the meanof data does not confirm to the normal distribution used tojustify the z-test. This is illustrated by the z-test on theentire data after outliers removal which has higher powerthan the bootstrap mixture SPRT, as expected from a testwith an apriori access to the entire data.8 The robustnessof the bootstrap mixture SPRT to outliers comes from twofactors: (i) the use of the bootstrap for learning the likelihoodwhich typically works better for the heavy tail distributions;(ii) working with blocks of data confines the effect of extremevalues in the data in a block to that block only by making thelikelihood ratio for that block close to one without affectingthe other blocks. Thus, not only the bootstrap mixtureSPRT allows continuous monitoring, it is more robust to themisspecification in the distribution generating the data.

Figure 4 shows the average duration of the test usingthe bootstrap SPRT. The test duration is the numbers ofsamples consumed in each split until either all the data isexhausted or the null hypothesis is rejected. We take theaverage over the 250 random splits of the data set. Notice

8Identifying outliers is a complicated problem by itself, oftenwith no good agreement on the definition of outliers. Here,we simply clip the values to some upper limit affecting onlya negligible portion of data to show the effect of extremevalues in data on the z-test.

614

0.00.2 0.4 0.6

0.8 1.0Theoretical Quantiles0.0

0.2

0.4

0.6

0.8

1.0Sa

mpl

e Q

uant

iles

query success rate per user session

0.00.2 0.4 0.6

0.8 1.0Theoretical Quantiles

Sam

ple

Qua

ntile

s

revenue per user session

(a) p-values for block size 10,000.

0.00.2 0.4 0.6

0.8 1.0Theoretical Quantiles 0.0

0.2

0.4

0.6

0.8

1.0

Sam

ple

Qua

ntile

s

query success rate per user session

0.00.2 0.4 0.6

0.8 1.0Theoretical Quantiles

Sam

ple

Qua

ntile

srevenue per user session

(b) p-values for block size 20,000.

Figure 2: Q-Q plot of p-values under bootstrap mixtureSPRT with different block sizes.

that the null hypothesis is rejected quickly for large changes,unlike the fixed sample size test where one has to wait untilthe precomputed number of samples have been collected.This avoids wasteful data collection and unnecessary wait incase of misspecification in the minimum effect size used todetermine the number of samples to be collected for fixedsample size tests.

In our experiments, we observed the power of the testswith smaller block size to be either comparable or better thanthe power of the tests with larger block size; this is in-linewith the observations made earlier about the advantage ofthe small block size. Similarly, the tests with smaller blocksize had smaller test duration.

Comparison with MaxSPRT on synthetic data: Fi-nally, we compare the performance of the bootstrap mixtureSPRT with MaxSPRT. We use synthetic data of size 100, 000generated from the Bernoulli distribution with success prob-ability equal to 0.05. The rejection threshold for MaxSPRTis determined by multiple A/A tests on synthetic datasetssuch that the type 1 error is close to 0.05, as described in [9].We also use A/A tests to determine the block size and thevariance of the Gaussian prior for the bootstrap SPRT suchthat the type 1 is close to 0.05. This resulted in the blocksize of 1000. Figure 5a shows comparable test power forMaxSPRT and the bootstrap mixture SPRT with MaxSPRTperforming marginally better. The better performance ofMaxSPRT is expected; here the data distribution is exactlyknown and is exploited by MaxSPRT. The bootstrap mixtureSPRT, however, learns the distribution in a nonparametricway and yet performs comparable to MaxSPRT. A simi-

0 2 8 104 6

true difference in percent0.0

0.2

0.4

0.6

0.8

1.0

pow

er

query success rate per user session

z-test fixed samplesbootstrap mixture sprt

(a) test power for query success rate per user session.

0 2 8 104 6

true difference in percent0.0

0.2

0.4

0.6

0.8

1.0

pow

er

revenue per user session

z-test fixed samples bootstrap mixture sprt z-test outliers removed

(b) test power for revenue success rate per user session.

Figure 3: Power comparison for z-test with fixed sample sizeand the bootstrap mixture SPRT.

lar observation can be made for the average test duration:MaxSPRT has slightly smaller test duration as shown inFigure 5b. MaxSPRT updates its likelihood ratio score foreach data point as it arrives. By contrast the bootstrapmixture SPRT updates its likelihood ratio score in blocksof data which contributes to a slightly longer average testduration. Block update, however, is a more realistic scenarioin a practical system where data usually arrives in blocks ofsize few hundreds to a few thousand. The general purposenature of the bootstrap mixture SPRT while still performingcomparable to MaxSPRT which requires exact knowledge ofthe data distribution is appealing.

6. CONCLUSIONSWe propose the bootstrap mixture SPRT for the hypothesis

test of complex metrics in a sequential manner. We highlightthe theoretical results justifying our approach and provideempirical evidence demonstrating its virtues using real datafrom an e-commerce website. The bootstrap mixture SPRTallows the flexibility of continuous monitoring while control-ling the type 1 error at any time, has good power, is robustto misspecification in the distribution generating the data,and allows quick inference in case the true change is largerthan the estimated minimum effect size. The nonparametricnature of this test makes it suitable for the cases where the

615

0 2 8 104 6

true difference in percent0

200000

400000

600000

800000

1000000av

g du

ratio

n

test avg duration for the bootstrap mixture SPRT

query success rate per user session

revenue per user session

Figure 4: Test average duration of the bootstrap mixture SPRTfor different metrics.

data generation distribution is unknown or hard to model,which typically is the case.

7. REFERENCES[1] T. Crook, B. Frasca, R. Kohavi, and R. Longbotham. Seven

pitfalls to avoid when running controlled experiments on theweb. In Proceedings of the 15th ACM SIGKDD InternationalConference on Knowledge Discovery and Data Mining, KDD’09, pages 1105–1114, New York, NY, USA, 2009. ACM.

[2] A. C. Davison, D. V. Hinkley, and B. Worton. Bootstraplikelihoods. Biometrika, 79(1):113–130, 1992.

[3] B. Efron. Bayes and likelihood calculations from confidenceintervals. Biometrika, 80(1):3–26, 1993.

[4] B. Efron and R. Tibshirani. An Introduction to the Bootstrap.Chapman & Hall/CRC Monographs on Statistics & AppliedProbability. Chapman & Hall/CRC, 1993.

[5] M. Goodson. Most winning a/b test results are illusory.Whitepaper, Qubit, Jan 2014.

[6] P. Hall. On the bootstrap and likelihood-based confidenceregions. Biometrika, 74(3):481–493, 1987.

[7] P. Hall. The Bootstrap and Edgeworth Expansion. SpringerSeries in Statistics. Springer, Mar 1997.

[8] R. Johari, L. Pekelis, and D. J. Walsh. Always valid inference:Bringing sequential analysis to a/b testing.http://arxiv.org/abs/1512.04922, Feb 2016.

[9] E. Kharitonov, A. Vorobev, C. Macdonald, P. Serdyukov, andI. Ounis. Sequential testing for early stopping of onlineexperiments. In Proceedings of the 38th International ACMSIGIR Conference on Research and Development inInformation Retrieval, SIGIR ’15, pages 473–482, New York,NY, USA, 2015. ACM.

[10] R. Kohavi, R. Longbotham, D. Sommerfield, and R. M. Henne.Controlled experiments on the web: Survey and practical guide.Data Min. Knowl. Discov., 18(1):140–181, Feb. 2009.

[11] M. Kulldorff, R. L. Davis, M. Kolczak†, E. Lewis, T. Lieu, andR. Platt. A maximized sequential probability ratio test for drugand vaccine safety surveillance. Sequential Analysis,30(1):58–78, 2011.

[12] T. L. Lai. Sequential analysis: Some classical problems and newchallenges. Statistica Sinica, 11(2):303–351, 2001.

[13] E. Lehmann. Elements of Large-Sample Theory, chapter 6,pages 363–450. Springer Texts in Statistics. Springer, Aug 2004.

[14] L. Pekelis, D. Walsh, and R. Johari. The new stats engine.Whitepaper, Optimizely, Jan 2015.

[15] S. J. Pocock. Group sequential methods in the design andanalysis of clinical trials. Biometrika, 64(2):191–199, 1977.

[16] M. Pollak. Optimality and almost optimality of mixturestopping rules. The Annals of Statistics, 6(4):910–916, 1978.

[17] H. Robbins. Statistical methods related to the law of theiterated logarithm. The Annals of Mathematical Statistics,41(5):1397–1409, 1970.

[18] J. P. Simmons, L. D. Nelson, and U. Simonsohn. False-positivepsychology: undisclosed flexibility in data collection and

0 2 8 104 6

true difference in percent0.0

0.2

0.4

0.6

0.8

1.0

pow

er

test power comparison

bootstrap mixture SPRT MaxSPRT

(a) Test power for the bootstrap mixture SPRT and MaxSPRT.

0 2 8 104 6

true difference in percent0

20000

40000

60000

80000

100000

avg.

dur

atio

n

test avg. duration comparison

bootstrap mixture SPRTMaxSPRT

(b) Average test duration for the bootstrap mixture SPRT andMaxSPRT.

Figure 5: Comparison of the bootstrap mixture SPRT withMaxSPRT.

analysis allows presenting anything as significant. Psychol Sci,22(11):1359–1366, Nov 2011.

[19] G. M. van Kempen and L. J. van Vliet. Mean and variance ofratio estimators used in fluorescence ratio imaging. Cytometry,39(4):300–305, Apr 2000.

[20] A. Wald. Sequential tests of statistical hypotheses. The Annalsof Mathematical Statistics, 16(2):117–186, 1945.

616