bootstrap methods for standard errors

8/13/2019 Bootstrap Methods for Standard Errors

1/23

Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of

Statistical AccuracyAuthor(s): B. Efron and R. TibshiraniSource: Statistical Science, Vol. 1, No. 1 (Feb., 1986), pp. 54-75Published by: Institute of Mathematical StatisticsStable URL: http://www.jstor.org/stable/2245500

Accessed: 19/10/2010 11:46

Your use of the JSTOR archive indicates your acceptance of JSTOR's Terms and Conditions of Use, available at

http://www.jstor.org/page/info/about/policies/terms.jsp. JSTOR's Terms and Conditions of Use provides, in part, that unless

you have obtained prior permission, you may not download an entire issue of a journal or multiple copies of articles, and you

may use content in the JSTOR archive only for your personal, non-commercial use.

Please contact the publisher regarding any further use of this work. Publisher contact information may be obtained at

http://www.jstor.org/action/showPublisher?publisherCode=ims.

Each copy of any part of a JSTOR transmission must contain the same copyright notice that appears on the screen or printed

page of such transmission.

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of

content in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms

of scholarship. For more information about JSTOR, please contact [email protected].

Institute of Mathematical Statisticsis collaborating with JSTOR to digitize, preserve and extend access to

Statistical Science.
http://www.jstor.org/action/showPublisher?publisherCode=imshttp://www.jstor.org/stable/2245500?origin=JSTOR-pdfhttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/action/showPublisher?publisherCode=imshttp://www.jstor.org/action/showPublisher?publisherCode=imshttp://www.jstor.org/page/info/about/policies/terms.jsphttp://www.jstor.org/stable/2245500?origin=JSTOR-pdfhttp://www.jstor.org/action/showPublisher?publisherCode=ims


2/23

Statistical cience1986,Vol. 1, No. 1, 54-77BootstrapMethodsfor tandardErrors,Confidencentervals,ndOtherMeasuresofStatisticalAccuracyB. Efron nd R. Tibshirani

Abstract.This is a review fbootstrapmethods,oncentratingn basicideas ndapplicationsatherhan heoreticalonsiderations.tbeginswithan exposition f thebootstrapstimate fstandard rror or ne-samplesituations.everal xamples,ome nvolvinguitecomplicatedtatisticalprocedures,regiven. hebootstrapsthen xtendedoothermeasuresfstatisticalccuracyuch as bias andpredictionrror,ndto complicateddatastructuresuch as time eries, ensored ata,andregression odels.Severalmore xamplesrepresentedllustratinghesedeas.The lastthirdof hepaperdeals mainly ith ootstraponfidencentervals.Keywords: Bootstrapmethod, stimatedtandard rrors,pproximateconfidencentervals, onparametricethods.

1. INTRODUCTIONA typical roblemn applied tatisticsnvolvesheestimationf nunknownarameter.The twomainquestions skedare (1) whatestimator shouldbeused? (2) Having hosen o use a particular , howaccurates it as an estimatorf0? The bootstraps ageneralmethodologyor nsweringhesecond ues-tion. t sa computer-basedethod, hich ubstitutesconsiderablemountsfcomputationnplaceof he-oretical nalysis.As we shallsee,thebootstrap anroutinelynswer uestionswhich re far oo compli-catedfor raditionaltatisticalnalysis. venfor el-atively imple roblemsomputer-intensiveethodslike hebootstrapre an increasinglyooddataana-lytic argainn an era of xponentiallyecliningom-putationalosts.This paperdescribes he basis of the bootstraptheory, hich svery imple,ndgives everal xam-ples of tsuse. Related deas like the ackknife,hedeltamethod,ndFisher'snformationound realso

discussed.Mostof heproofsndtechnical etails reomitted. hesecanbe foundn thereferencesiven,B. EfronsProfessorf tatisticsndBiostatistics,ndChairmanftheProgramn Mathematicalnd Com-putationalcience tStanford niversity.is mailingaddresssDepartmentf tatistics,equoiaHall,Stan-fordUniversity,tanford,A 94305.R. Tibshiranisa Postdoctoralellown theDepartmentfPreventiveMedicinendBiostatistics,aculty fMedicine, ni-versity f Toronto,McMurrick uilding,Toronto,Ontario, 5S 1A8,Canada.

particularlyfron 1982a). Some of the discussionhere s abridgedromfron nd Gong1983)and alsofrom fron1984).Before eginninghe mainexposition, e willde-scribe ow hebootstrap orks n terms f problemwhere t is not needed, ssessing heaccuracy f thesample mean.Supposethat our data consists f arandom amplefromn unknown robabilityistri-bution onthereal ine,(1.1) Xl X2, * ,X.- F.Having bserved 1 = x1,X2 = x2, .. , Xn = xn,wecomputehe samplemeanx = 1 xn/n, and wonderhowaccurate t is as an estimate f the truemean6 = EFIX}.Ifthe econd entralmomentfF is 182(F) EFX2- (EFX)2, then the standard error (F; n, x), that isthe tandard eviationf for sample f ize n fromdistribution, is(1.2) o(F) = [,M2(F)/n]112.

The shortenedotation (F) -(F; n, ) is allow-ablebecause he ample ize n andstatisticf nterestx are known, nlyF beingunknown. he standarderrors the traditionalmeasure f 's accuracy. n-fortunately,e cannot ctually se 1.2)toassess theaccuracyf , sincewedo notknowM2(F), butwecanusetheestimatedtandardrror(1.3) = [#2/n]l/2whereX2 Ei (Xi-x)2/(n - 1), theunbiased estimateofA2(F).There s a more bviousway oestimate (F). Let

54


3/23

BOOTSTRAP METHODS FOR MEASURES OF STATISTICAL ACCURACY 55F indicate he mpiricalrobabilityistribution,(1.4)F: probability ass 1/n n x1, 2, .. , xn.Thenwe can simply eplace byF in 1.2),obtaining(1.5) a - F) = [U2(P)/nl .as the estimated tandard rror or . This is thebootstrapstimate. he reasonfor he name"boot-strap"will e apparentn Section ,whenwe evaluatev(F) for tatisticsmore omplicatedhanx. Since(1.6) /2 a 82(F) = X (a is not quite he ameas a-,butthe differences toosmall obe importantnmost pplications.Of coursewe do not reallyneed an alternativeformula o (1.3) in this case. The troublebeginswhenwe want standardrror or stimators ore ompli-catedthanx,for xample, median r a correlationor slope oefficientrom robust egression.nmostcases there s no equivalent o formula1.2),whichexpresseshe tandard rror(F) as a simple unctionofthe sampling istribution. As a result, ormulaslike 1.3) do not xistformost tatistics.This is where hecomputeromes n. It turns utthatwecanalways umericallyvaluate hebootstrapestimate = a(F), without nowing simple xpres-sionfor (F). The evaluationf is a straightforwardMonteCarlo xercise escribednthenext ection.na good computingnvironment,s describedn theremarksn Section2, the bootstrap ffectivelyivesthe statistician simpleformulaike (1.3) for nystatistic,omatter ow omplicated.Standard rrors re crudebutusefulmeasures fstatisticalccuracy. heyarefrequentlysed togiveapproximateonfidencentervals oran unknownparameter(1.7) 0 E 0 ? Sz(a),where(a) sthe 100* a percentileoint f standardnormal ariate, .g., (95) = 1.645. nterval 1.7) issometimesood, nd sometimesot ogood. ections7 and 8 discuss more ophisticatedseofthe boot-strap,which ivesbetter pproximateonfidencen-tervalshan 1.7).The standard nterval1.7) is basedon takingit-erally he large amplenormal pproximationf -0)/S N(0, 1). Applied tatisticiansse a varietyftricks o improvehis pproximation.or nstancef0 is the correlationoefficientnd0 thesample or-relation,hen he transformation= tanh-1(0), =tanh-1(0) reatlymproveshenormalpproximation,at least nthose ases where heunderlyingamplingdistributions bivariatenormal.The correct acticthen s to transform,ompute he nterval1.7) forand transformhis nterval ackto the0scale.

Wewill ee thatbootstraponfidencentervalsanautomaticallyncorporatericks ike his,without e-quiringhedata analyst o produce pecial echniques,like he anh-1 ransformation,or achnew ituation.An mportanthemefwhat ollowss the ubstitutionofraw omputingower or heoreticalnalysis. hisis not an argument gainst heory,fcourse, nlyagainst nnecessaryheory.Most common tatisticalmethods ere evelopednthe1920s nd1930s,whencomputationas slow ndexpensive. ow that om-putations fast nd cheapwe canhopefor ndexpectchanges n statisticalmethodology.his paperdis-cussesone suchpotential hange, fron1979b)dis-cusses everal thers.2. THE BOOTSTRAP ESTIMATEOF STANDARDERRORThis section resents more areful escriptionfthebootstrapstimatef tandardrror. or nowwewillassume hattheobserved atay = (xl, x2, **xn)consists f ndependentnd denticallyistributed(iid) observations 1,X2- .-, Xn fiid , as in (1.1).Here F representsn unknown robabilityistribu-tionon r, thecommon sample space of the observa-tions. We have a statistic f interest, ay 0(y), towhichwe wish o assign n estimatedtandard rror.Fig. 1 shows n example.The sample pace r is

n2+, the positive uadrant f the plane.We haveobserved = 15 bivariate ata points, ach corre-sponding o an Americanaw school.Each pointxiconsists f wo ummarytatisticsor he1973 nter-ing lassat law school(2.1) xi= (LSATi, GPAi);LSATi is the class' average coreon a nationwideexamcalled"LSAT"; GPAi s the class' average n-dergraduaterades. heobserved earson orrelation

3.5-GPA *1

3.3t *23.1 -10 *6GPA - 97 *42.9 - * 15@1403@13 *122.7 I- l I540 560 580 600 620 640 660 680LSAT

FIG. 1. The law schooldata (Efron, 979b).The datapoints,begin-ning with School 1, are (576, 3.39), (635, 3.30), (558, 2.81),(578, 3.03), (666, 3.44), (580, 3.07), (555, 3.00), (661, 3.43),(651, 3.36), (605, 3.13), (653, 3.12), (575, 2.74), (545, 2.76),(572, 2.88), (594, 2.96).


4/23

56 B. EFRON AND R. TIBSHIRANIcoefficientor hese15 points s 6 = .776.We wish oassign standardrror o this stimate.Let o(F) indicate he standard rror f 0, as afunctionf heunknownampling istribution,(2.2) a(F) = [VarF{Ny)Ofcourse F) is also a functionf he ample ize nand theform f thestatistic (y), butsincebothofthese are known heyneednot be indicatedn thenotation. he bootstrap stimate f tandard rrors(2.3) =where is the empirical istribution1.4), puttingprobability/n neach observed ata point i. n thelawschool xample, is thedistributionuttingmass1/15on each point n Fig. 1, and a is the standarddeviation f he orrelationoefficientor 5 idpointsdrawn rom .In most cases, includinghat of the correlationcoefficient,here sno simple xpressionor hefunc-tiona(F) in (2.2).Nevertheless,t s easyto numeri-cally valuate = 0(F) bymeansof a Monte Carloalgorithm,hich epends n the following otation:= (x4,4, *, x*) indicates independentrawsfrom , called a bootstrapample.Because F is theempiricalistributionf hedata, bootstrapampleturns ut to be thesameas a random ample f sizen drawnwithreplacementrom he actual sample{X1, X2, .. * * Xnl.The MonteCarlo lgorithmroceedsn three teps:(i) using random umber enerator,ndependentlydraw largenumber fbootstrapamples, ay y*(1),y*(2), ***,y*(B); (ii) for ach bootstrap ample y*(b),evaluate he tatistic f nterest,ay 0*(b)= 0(y*(b))gb= 1,2, * , B; and (iii) calculatethesamplestandarddeviationf he0*(b)values

-A 0()2A 1/2Zb=1 {8*(b)- *.)}0/(2.4) B-i

l*(.)= >20*(b)B~~~~~~It is easyto see thatas B - 60, 5B will approach

a = (F), thebootstrapstimate f standard rror.Allwe are doing s evaluating standard eviationby MonteCarlo sampling. ater, n Section9, wewilldiscusshowlargeB need be taken.For mostsituations intherange 0to200 s quite dequate.In whatfollows e willusually gnore hedifferencebetweenB anda, calling oth implya"Why s eachbootstrapample akenwith he amesample izenas theoriginal ataset? Rememberhato(F) is actually F, n, 6), thestandard rror or hestatistic( ) basedona randomample f ize n fromthe unknown istribution. The bootstrapstimatefis actually (F, n,0) evaluated t F = F. The Monte

Carlo lgorithmillnot onvergeoa' f hebootstrapsample ize differsrom he ruen. Bickel nd Freed-man 1981) showhow o correcthe lgorithmo givea if nfact hebootstrapample ize s taken ifferentthann, but so fartheredoes not seemto be anypractical dvantage o be gained n thisway.Fig.2 shows he histogramf B = 1000bootstrapreplicationsf he orrelationoefficientrom he awschooldata. For convenient eferenceheabscissa splotted n termsof0* 0 = 0* - .776. Formula (2.4)gives6' = .127 as the bootstrap stimate fstandarderror. his can be comparedwith he usual normaltheory stimatef tandard rror or ,(2.5) TNORM= (1 - )/(n - 3)1/ = .115,[Johnsonnd Kotz 1970, . 229)].

REMARK.The Monte Carlo algorithmeading oU7B2.4) ssimple oprogram.n theStanford ersionof the statistical omputinganguageS, ProfessorArthurwenhas ntroduced single ommand hichbootstrapsny tatisticntheS catalog. or nstancethebootstrapesultsnFig.2 areobtained imply ytyping

tboot(lawdata,orrelation, = 1000).The executionime s about factor fB greaterhanthatfor heoriginalomputation.

There is anotherway to describe he bootstrapstandardrror: is thenonparametricaximumike-lihood stimateMLE) of heunknownistribution(Kiefer nd Wolfowitz,956).This means that thebootstrap stimate f= a(F) is the nonparametricMLE ofv(F), the rue tandardrror.In fact here s nothing hich ays thatthe boot-strapmust ecarriedutnonparametrically.upposefor nstance hat nthe aw school xamplewe believethe truesampling istribution mustbe bivariatenormal. hen wecould stimate with tsparametricMLE FNORM, thebivariate ormal istributionavingthesamemeanvector nd covariancematrix s the

NORMALTHEORYHITGADENSITY HISTOGRAM

HISTOGRAMPERCENTILES160/o 50 84-0.4 -0.3 -0.2 -0.1 0 0.1 0.2

FIG. 2. Histogram fB = 1000 bootstrap eplications f6*forthelaw schooldata. The normal heory ensity urvehas a similar hape,butfallsoff ore uickly t the upper ail.


5/23

BOOTSTRAP METHODS FOR MEASURES OF STATISTICAL ACCURACY 57data. The bootstrapamples t step i) ofthe algo-rithm ould henbe drawn rom NORMnstead fF,and steps ii) and iii) carried utas before.The smooth urve n Fig. 2 showstheresults fcarryingutthis "normal heory ootstrap"n thelaw schooldata.Actuallyhere s no needto do thebootstrapamplingn thiscase,because of Fisher'sformulaor he amplingensity f correlationoef-ficientnthebivariate ormal ituationseeChapter32 ofJohnsonnd Kotz,1970).This densityan bethoughtof as the bootstrapdistribution or B = oo.Expression2.5) is a closeapproximationo "1NORMo(FNORM), theparametricootstrapstimate f tand-arderror.In consideringhe merits r demeritsftheboot-strap, t is worth ememberinghatall of the usualformulas orestimatingtandard rrors,ike g-l/2whereJ is the observedFisher information,re es-sentially ootstrap stimates arried ut in a para-metricramework.hispoint s carefullyxplainednSection ofEfron1982c).The straightforwardon-parametriclgorithmi)-(iii)has thevirtuesf void-ingall parametricssumptions,ll approximations(suchas those nvolvedwith he Fisher nformation

TABLE 1A sampling xperimentomparing hebootstrapnd ackknifeestimates f tandard rror or he25% trimmedmean,samplesize n = 15F standard F negativenormal exponential

Ave SD CV Ave SD CVBootstrap f .2s87 .071 .25 .242 .078 .32(B = 200)Jackknifef .280 .084 .30 .224 .085 .38True minimumV) .286 (.19) .232 (.27)

expressionor he tandardrror f n MLE), and nfactall analytic ifficultiesf anykind. The dataanalyst s free o obtainstandard rrors orenor-mously omplicatedstimators,ubjectonly to theconstraintsf omputerime. ections and 6discusssome nterestingppliedproblems hich re far oocomplicatedor tandardnalyses.How welldoes the bootstrap ork? able 1 showsthe answern one situation. ere r is the real ine,n = 15, and the statistic of interests the 25%trimmed ean. f thetrue ampling istributionisN(0, 1),then hetrue tandard rrors a(F) = .286.The bootstrapstimatea s nearlynbiased,veraging.287in a large sampling xperiment.he standarddeviationfthe bootstrapstimate ' s itself071 nthis ase,with oefficientfvariation071/.287 .25.(Noticethat thereare two levelsof MonteCarloinvolvednTable 1: first rawingheactual amplesy = (xl, x2, .., x15)from , and thendrawingoot-strapsamples (x4, x2*, **, x15)withy heldfixed.Thebootstrapamples valuate ' for fixed alue of y.The standard eviation071refers o thevariabilityof a' due to therandom hoice fy.)The ackknife,notherommonmethod f assign-ingnonparametrictandard rrors,s discussed nSection10.The jackknifestimate J' is also nearlyunbiased or (F), buthashigheroefficientf vari-ation CV). The minimumossibleCV for scale-invariant stimate fa(F), assuming ullknowledgeof theparametricmodel, s shown n brackets. henonparametricootstraps seen to be moderatelyefficientn both asesconsiderednTable 1.Table 2 returnso the case of 0 the correlationcoefficient.nsteadof real data we havea samplingexperimentn which hetrueF is bivariate ormal,true orrelation= .50, sample izen = 14. Table 2is abstractedrom larger able n Efron1981b), n

TABLE 2Estimatesof tandard rror or he correlationoefficientand forX = tanh-'6; samplesize n 14, distribution bivariatenormalwith ruecorrelation = .5 (from larger able n Efron, 981b)Summary tatisticsfor200 trials

Standard error stimatesforC Standarderror stimatesforXAve SD CV MSE Ave SD CV -VKfM

1. Bootstrap B = 128 .206 .066 .32 .067 .301 .065 .22 .0652. BootstrapB = 512 .206 .063 .31 .064 .301 .062 .21 .0623. NormalsmoothedbootstrapB = 128 .200 .060 .30 .063 .296 .041 .14 .0414. Uniform moothedbootstrapB = 128 .205 .061 .30 .062 .298 .058 .19 .0585. Uniform moothedbootstrapB = 512 .205 .059 .29 .060 .296 .052 .18 .0526. Jackknife .223 .085 .38 .085 .314 .090 .29 .0917. Delta method .175 .058 .33 .072 .244 .052 .21 .076(Infinitesimalackknife)8. Normal theory .217 .056 .26 .056 .302 0 0 .003True standarderror .218 .299


6/23

58 B. EFRON AND R. TIBSHIRANIwhich ome f he methodsor stimatingstandarderror equiredhe ample ize to beeven.The left ide of Table 2 refers o0, while he rightside refers o X = tanh-'(0) = .5 log(1 + 6)/(1 -).For eachestimator fstandard rror,he rootmeansquared rror f estimationE(a - )2]1/2 iS given nthecolumn eadedVMi..Thebootstrap asrunwithB = 128 and alsowithB = 512, he atter alueyielding nly lightly etterestimatesnaccordance ith heresults fSection .Furtherncreasing wouldbe pointless. t can beshown hatB = oogivesVii-i = .063for ,only001lessthanB = 512.Thenormal heory stimate2.5),whichwe know o be idealfor his sampling xperi-ment, as ../i-Si .056.We can compromiseetween hetotally onpara-metricootstrapstimate 'andthe otally arametricbootstrapstimate7NORM. This is done n lines3, 4,and 5 of Table 2. Let 2 = Sin-l (xi - )(x- i)'/n bethesample ovariancematrix f the observed ata.The normalmoothedootstrapraws he bootstrapsamplefrom (DN2(0, 252), (Dindicating onvolu-tion.Thisamountsoestimating byan equalmix-tureofthe n distributions2(xi, 252), that s byanormal indow stimate. ach point i*n a smoothedbootstrapample s the sumof a randomlyelectedoriginal atapointxj,plusan independentivariatenormal oint j - N2(0, 252). Smoothing akes ittledifferencen the eft ideofthetable, ut s spectac-ularly ffectiven the k case. The latterresult ssuspect ince he true ampling istributions bivar-iate normal,nd the function = tanh- is specifi-cally hosen ohavenearlyonstant tandardrrornthe bivariate ormal amily. he uniformmoothedbootstrap amples from F (D W(0, .252), whereWI(0, 252) is the uniform istributionn a rhombusselectedoVIhas meanvector and covariancematrix.25Z. tyieldsmoderate eductionsnvMi-SRor othsidesof he able.Line6 ofTable2 refers o the deltamethod, hichis the mostcommonmethod fassigning onpara-metrictandardrror.urprisinglynough,t sbadlybiaseddownwardn both idesof he able.The deltamethod,lso known s themethod fstatistical if-ferentials,heTaylor eriesmethod,ndthe nfinites-imal ackknife,s discussedn Section 0.

3. EXAMPLESExample 1. Cox's ProportionalHazards Model

In this sectionwe applybootstraptandard rrorestimationo some omplicatedtatistics.The data for hisexample omefrom study fleukemia emissionimes n mice, aken fromCox(1972). Theyconsist f measurementsf remission

time y) inweeksfor wo groups, reatmentx = 0)and control x = 1), and a 0-1 variable (bi) indicatingwhether r not the remissionime s censored0) orcomplete1). There re21mice neach group.Thestandard egression odel or ensored ata sCox's proportionalazardsmodel Cox, 1972). t as-sumes hat hehazard unction(t x),theprobabilityofgoing nto remissionn next nstant ivenno re-mission pto time for mousewith ovariate , sof heform(3.1) h(t x) = ho(t)e:x.Hereho(t) s an arbitrarynspecifiedunction.incex here s a group ndicator,hismeans implyhat hehazard or he ontrol roupse: times hehazard orthe treatmentroup. he regression arameter isestimatedndependentlyfho(t)throughmaximiza-tion f he o called partialikelihood"

e,3xi(3.2) PL = 11 e-xiiED EiER, e i'where is the set of ndices fthe failure imes ndRi s the et of ndices f hose t risk t timeyi.Thismaximizationequiresn iterativeomputerearch.The estimate for hesedata turns ut to be 1.51.Taken iterally,his aysthat he hazardrate s e'5'= 4.33timeshighernthe control roup han nthetreatmentroup, o the treatments very ffective.What s the tandardrror ffA?he usual symptoticmaximumikelihoodheory,ne over hesquarerootof heobserved ishernformation,ives n estimateof 41.Despite hecomplicatedature fthe estima-tion rocedure,ecan also estimatehe tandard rrorusingthe bootstrap.We samplewithreplacementfromhetriples (y', xi, 5k), *.*, (Y42, x42,642)). Foreach bootstrapsample $(y*, x*, 0), .., (y*, x4*2,6*)) weformhepartial ikelihood nd numericallymaximizet toproduce hebootstrapstimate *.Ahistogramf1000bootstrapalues s shownnFig.3.The bootstrapstimate fthe standard rror fAbased on these 1000 numberss .42. Althoughhebootstrapnd standard stimatesgree,t s interest-ingtonote hat hebootstrap istributions skewedto the right.This leads us to ask: is there otherinformationhatwe can extract rom hebootstrapdistributionther han standard rrorstimate?heanswers yes-in particular,hebootstrap istribu-tioncanbe used to form confidencenterval or A,aswe will ee n Section . Theshapeof hebootstrapdistributiionill help determinehe shape of theconfidencenterval.In this xample urresamplingnitwas thetriple(yi,xi,bi), ndwe gnoredheunique lementsftheproblem,.e., hecensoring,ndtheparticularmodelbeing sed. n fact, here re otherways obootstrap


7/23

BOOTSTRAP METHODS FOR MEASURES OF STATISTICAL ACCURACY 59200

150

100

50

010.5 1 1.5 2 2.5 3FIG. 3. Histogram of 1000 bootstrapreplications or the mouseleukemiadata.thisproblem.Wewill eethiswhenwe discuss oot-strappingensored ata nSection .Example : Linear ndProjection ursuitRegression

We illustraten application f the bootstrap ostandardinear eastsquaresregressions well s toa nonparametricegressionechnique.Consider he tandard egressionetup.Wehavenobservationsna response and covariatesX1,X2,* ,X,). Denote he thobserved ector fcovariatesbyxi = (xil,xi2, .. , xip)'.The usual linearregressionmodel ssumesp(3.3) E(Yi) = a + E /fl1xi.j=l

Friedmannd Stuetzle1981) ntroduced more en-eralmodel,he rojection ursuitregression odelm(3.4) E(Yi)= X sj(aj - xi).j=l

Thep vectors j areunitvectors"directions"),ndthefunctionsj(.) areunspecified.Estimation of la,, sl(.)), ..., {am1, m(-)} is per-formedna forwardtepwisemanners follows. on-sider al, s ( - . Givena direction l, s, *) is estimatedby nonparametricmoothere.g., unningmean)ofy on a, *x. The projection ursuit egressionlgo-rithm earches ver all unit directionso find hedirection l and associated function l(-) that mini-mize sy1-sl(i *a))2. Then residuals are takenandthe nextdirectionnd functionre determined.This process s continued ntilno additional ermsignificantlyeducesheresidual umof quares.

Notice herelation f heprojectionursuit egres-sionmodel o the standard inearregression odel.When he functionj(-) is forced o be linear nd isestimated y the usual least squaresmethod, one-term rojection ursuitmodel s exactly he sameasthe standard inear egression odel.That is to say,the fittedmodel '(a'1 xi) exactly quals the leastsquaresfit ' + f31xi,.This is becausetheleastsquares it, ydefinition,inds hebestdirectionndthe best inearfunction f thatdirection. ote alsothat dding notherinear erm 2(& * X2)wouldnotchange he fittedmodel ince the sum of two inearfunctionss anotherinear unction.Hastieand Tibshirani1984) applied hebootstrapto the inear ndprojectionursuit egression odelsto assess thevariabilityf the coefficientsn each.The datathey onsideredre aken romreiman ndFriedman1985).The responseY is Uplandatmos-pheric zoneconcentrationppm);thecovariates 1= SandburgAirForce base temperatureCO),X2 =inversionase heightft),X3 Daggotpressure ra-dientmmHg),X4 = visibilitymiles), ndX5 = dayoftheyear.There re 330 observations.he numberof ermsm) n themodel3.4) s taken o be two.Theprojectionursuit lgorithmhose irectionsl = (.80,-.38, .37, -.24, -.14)' and 62= (.07, 16, 04, -.05,-.98)'. These directionsonsistmostly fSandburgAirForcetemperaturend dayof theyear,respec-

a5 I

a4 A. I

a3

a2

a, L-1 -0.5 0 0.5 1bootstrappedoefficients

FIG. 4. Smoothedhistograms fthebootstrappedoefficientsor hefirst erm n theprojection ursuit egressionmodel. olid histogramsarefor he usualprojection ursuitmodel; he dottedhistogramsrefor inear (*).


8/23

60 B. EFRON AND R. TIBSHIRANI

a4

a3

a2

al~~ ~ ~~~~~~ p p-1 -0.5 0 0.5 1bootstrapped coefficients

FIG. 5. Smoothedhistograms f hebootstrappedoefficientsor hesecond term n theprojection ursuit model.

tively.Wedo not howgraphs f heestimatedunc-tions (*(.) and s2(. althoughn a full nalysis fthedata theywould lso beof nterest.) orcing '( *) tobe linear esultsnthe direction' = (.90, -.37, .03,-.14, -.19)'. These are ust the usual leastsquaresestimates1, *.,* ,,scaled so thatEP2 = 1.To assess the variabilityfthedirections, boot-strap ample sdrawnwith eplacementromYi,x11,

. . ., X15), ***, (Y330, X3301,*- , X3305)andtheprojectionpursuit lgorithms applied.Figs.4 and5 showhis-togramsfthedirections'*anda* for 00 bootstrapreplications.lso shownnFig.4 (broken istogram)are the bootstrap replicationsof a& with .(.) forcedtobe linear.The first irectionfthe projection ursuitmodelis quite stableand only lightlymorevariable hanthe orrespondinginear egressionirection. uttheseconddirections extremelynstable t is clearlyunwise oput nyfaith n the second irectionf heoriginal rojection ursuitmodel.Example 3: Cox's Model and Local LikelihoodEstimation

In thisexample,we return o Cox's proportionalhazardsmodel escribednExample ,butwith fewadded wists.Thedata thatwewilldiscuss omefrom heStan-ford eart ransplantrogramnd aregivennMillerandHalpern 1982).The response is survival imeinweeks fter heart ransplant,he covariate isage at transplant,nd the 0-1 variable3 indicateswhetherhe survival ime s censored0) or complete

(1). There re measurementsn 157patients. pro-portional azardsmodelwasfit o thesedata,withquadratic erm, .e, h(t Ix) = ho(t)elx+i32x. Both#, ndf2 arehighlyignificant;hebroken urve nFig.6 is/3x f2x2s a functionfx.For comparison, ig. 6 shows solid ine) anotherestimate. his was computed sing ocal likelihoodestimationTibshirani nd Hastie, 1984). Given ageneral roportionalazardsmodel f heform (t x)= ho(t)es(x), he local likelihoodtechnique assumesnothingbout heparametricorm fs(x); insteadtestimates(x) nonparametricallysing kind f ocalaveraging.he algorithmsvery omputationallyn-tensive,nd tandardmaximumikelihoodheoryan-notbe applied.A comparison f the two functions eveals n im-portant ualitativeifference:heparametricstimatesuggests hat hehazarddecreases harply pto age34, thenrises; he ocal ikelihoodstimate tays p-proximatelyonstant ptoage45 then ises.Has theforced ittingf quadratic unctionroduced mis-leading esult? o answer his uestion, e can boot-strap he ocal likelihoodstimate.We samplewithreplacementrom hetriples (Yl, x1,61) .. (Y157iX157, 157) and apply he ocal ikelihoodlgorithmoeach bootstrapample. ig.7 shows stimated urvesfrom 0 bootstrapamples.Someofthecurves re flat ptoage 45,othersredecreasing. ence theoriginal ocal likelihood sti-mate shighly ariablen thisregion ndon thebasisofthesedatawecannotdeterminehe truebehaviorof hefunctionhere.A lookback t theoriginal atashows hatwhilehalf f the patientswereunder 5,only13% of thepatientswereunder 0. Fig. 7 alsoshows hat he stimates stablenear hemiddle gesbutunstable or heolder atients.

3

2

0 / ///~~~~~~~~~~\L //a /t

10 20 30 40 50 60age

FIG. 6. Estimatesof ogrelative iskfortheStanfordheart trans-plant data. Broken curve: parametric estimate. Solid curve: locallikelihood stimate.


9/23

BOOTSTRAP METHODS FOR MEASURES OF STATISTICAL ACCURACY 614

3

-11,..,.10 20 30 40 50 60age

FIG. . 20bootstrapsf he ocal ikelihoodstimateor he tanfordheartransplantata.

4. OTHER MEASURESOF STATISTICALERRORSo farwehave discussed tatisticalrror,raccu-racy,nterms f he tandard rror.t seasy o assessothermeasures f statistical rror,uchas bias orpredictionrror, sing hebootstrap.Consider he stimationfbias.For giventatistic0(y), nda given arameteru(F), et

(4.1) R(y,F) = 0(y) - A(F).(Itwillhelpkeep urnotationlear o calltheparam-eter f nterest rather han0.)Forexample,umightbe themean f hedistribution,assuminghe amplespaceX isthereal ine, nd0the25% trimmed ean.The bias of0for stimatingt s(4.2) A(F) = EFR(Y, F) = EF10(y)) - A(F).ThenotationF indicatesxpectation ith espectotheprobability echanismppropriateo F, in thiscase y = (xl, x2, - -*,xn)a randomsample from .Thebootstrapstimate fbias s(4.3), fi A(F) = EFR(y*, F) = Ep{0(Y*)) - U(F)As inSection , y*denotes randomample x4,x,***, ) from , i.e., bootstrapample.To numeri-callyevaluate 3, ll wedo is change tep iii) of thebootstraplgorithmn Section to

'A- 1 B/B - - E R(y*(b), F ).Bb=1(4.4) - b 0*(b) A-()

AsB-*oo,L 3goesto/F(4.3).

TABLE 3BHCGblood erumevels or 4 patients avingmetasticizedreastcancer nascendingrder

bootstrap methods for standard errors

Documents