understanding probability and statistical inference through resampling

From Brunelli, Lina & Cicchitelli, Giuseppe (editors). Proceedings of the First Scientific Meeting (of the IASE). Universiti di Perugia (Italy), 1994. Pages 199-21 1. Copyright holder: University of Perugia. Permission granted by Dipartmento di Scienze Statistiche to the IASE to make this book freely available on the Internet. This pdf file is from the IASE website at ht~:!!~~ww.stat.auckla~ld.nz!-iase!~ublica~ions!p-oc 1993. Copies of the complete Proceedings are available for 10 Euros from the IS1 (International Statistical Institute). See http:lljsi.cbs.n1isale-iase.htm for details.

UNDERSTANDING PROBABILITY AND STATISTICAL INFERENCE THROUGH RESAMPLING

Clifford Koriold Scientific Reasonirrg Resenrch Institute

Hnsbrourh Laboratory, Unzvrt-siiy of Mdssdchzae~s Amberst, MA 01003, USA

1. Introdr~ction

During tile past four years, I have been developing curricztlic and computer sofiware for teaclzing probability and data analysis at the itltroductory high-school and college level. The approach I've taken emphasizes thc use of red data, where "telling a story" ralces priority over testing hypotheses, and in which mathenlatical for~nalism is ltcpr to a minimum (see Cobb, 1992; Scheaffer, 1990; Watlcins ctltl., 1992).

A major question I have considered is how probability ought to be integrated into rhis n&w data-rich curriculum. There are two major reasons for keeping probability in the data-analysis (or statistics) curriculum. First, at some point in the process of constructing theories that accounr for patterns in data, it is important that students consider alterilative explanations. Among these is the possibility that some outcome of interest resulted from chance. Second, probability is an important concept in its own right (Falk and Konold, 1392). It comprises a world view and should not be viewed a necessary evil that must be faced if students are to understand statistical inference.

i n deciding how to relate probaLility and data at~al~sis, T have adopted an approach Julian Sitnon began &vacating in the late 1960s. Simon describes his approach as having grown out of his frustrariol~ watching graduate students do siily things when trying to test a staristical hypothesis (Simon and Bruce, 1992). He began designing experiments from which he hoped they could build up sound probnbitistic understandings. These eventually developed into a resampling approach that promised a nlore intuitive take oil probability and data analysis, and which made the connections between the two f elds more apparent. Of course, Simon didn't invent Mollre Carlo methods, nor the randomizatioo tests he would come ro employ, Lut he was among the first to see rheir educatiorlal potenrial, and long before the computer was widely available.

IASE/ISI Satellite, 1993: Clifford Konold

200 UNDERSTANDING PRORARILI?YAND S TATIS11CS THROUGH HESAMPUXC

Rather than elaborate Simon's argument here, I briefly describe two software tools we've developed, highlighting aspects rl-iat empilrtsize the relation between probability and data analysis. I also report some results from our primary test site, a high school in Hotyoke, R4assachusetts.

2. Modeling a problem with Prob Sim

Most educationai probability-simulation software con~prise several ready-made models (e.g., coin model, die model). Snlde~zts load the appropriate model, draw samples, and t11cn see results displayed. The software thus offers empirical dc~nolistrarions of various facts and principles, such as the Iaw of large numbers and the binomial distribution. The software we have designed, "Prob Sim", inclrrdes no ready-made models. Rarl~er, the student must build the model, specifying the appropriate sampling procedure and analyses in order ro estimate the probabiiity of some went. The process of buildi~xg a simulation nlodel is at least as important as, if not niore important rhan, drawing the appropriate conclusions fronl rhe resdrs. T o illustrate, I'll describe one of our activities entitled "LAPD" (see Xonold, 1933, for another example).

Students begin by reading excerpts from The Nezv Yoi-k Tinzes account (March 18, 1991) of the beating of Rodney King by officers of the Los Angeles Police Department. After reporting rlzat. "at least 15 officcrs in patrol cars converged on King", the article broaches one of the issues that made chis incident explosive: "In what other police officers called a chancc deployment, all the pursuing officers were white. Tile force, which nurnbcrs about 8,300, is 14% black?.

Scudcnts are asked to build a model of tliis situation LO estimate the probability of finding no blaclrs in a random sample of I6 officers. 'Yhis problem has generated lively discussions in our test sites. Wlzetz scude~lts care deeply about a problem, they are more willing to persist through difficulties. hbreover, they learn that applicarion of probability theory is not limited to rolling dice arid blindly clrawing soclcs out of dresser drawers.

2.1 Building a madel

I demonstrate below the various stages through which students progress in modeling this situation, illustracing the steps wid1 screen shots from f'rob Sim. In Fig, 1, a mixer has been filled according to rile irrforznatiori prot.ided in the article. There are a total of 8,300 elements, 1,162 of them labeled B (Black) and 7,138 labeled N (Not blaclt). The non-replacement option has been selected to preclude the possibility of having the sanic


C. KONOI D

clement (officer) in a sample more than once.

Figure 1. A snnplzi?g t~1odeIf.r the Isil'Dpro6br~t

The Run controls on the far left shows the sample size set at 16, and numbcr of repetitions set (somewhat arbitrarily) at 100. After tlte llun button is pressed, the cornputer draws I6 elements from the mixer without replacement, repeating chis a rota1 of 100 rimes. This is analogous to lookiilg ar 100 occasions when IG randomly-selected officers appeared on a crime scene. The sampling process is animated in the lower palt of the Mixer window. The results of each repetition arc displayed in a Data Record window (not shown).

After the block of 100 repetitions has been drawn, results can be analyzed as shown in Fig. 2. The Analysis window in this case shows the numbcr (and proportion) of occurrences of each of the 17 possible unordered outcomes. Ten percent of the samples had 0 B's. Thus, a first estimate of the probabjIiry that a "chance deployment" would include no black police of&cers is . 10.

2.2 Repeati~y the txperivzent

111 many textbook examples of simulation, the experimental component ends here. But i t is important not only to esrinlate the probability of sonic

event, bur to get sonie sense of [he range (or variability) in that estimate given the number of reperitions. Indeed, the notion of chance is nor


apparent iiz a value per se, but in the variability of result over

& File Edit Run flnalyze Windows

B B B B B B B B B B B B B B B N 0 B B B B B B B B 8 B B B B B N N O B B B B B E B B ' B B B B S N N N 0 B B B B B B B B B B B B N R N N o

B B B B B B B r f N N N N N t { N N 0 B B B B B B N N N N N N N N N N 0 B B B B B N N N N N N N N N N N 2

Figure 2. ?%r 1rtln4ih zutndow shozc!ing rhc ~rszilts of 100 rtprt/tlot.zi

replications of an experi~nenr. To replicate ail cxprrimer~t irl l'rob Sim, thc studcnr has only to press the run button again. Another random sample is dmwn, and rbe Analysis window updates to show the new results.

Students replicate tlte experiment about 10 times, plorting the resulrs olt a line plot whicll helps to emphasize the variability in the process. They &en pool their results to come up wid1 a final probability estimate. For the 10 resulrs plot tedi~~ Kg. -3, the pooIed esrinare of P (0 b~aclcss) is . I 09.

x X X X

X X X X X X

5 6 7 8 9 10 11 12 13 14 15 IG 17 Percentage of samples with no BS

Figure 3. Lincpht of reszrlt of10 rrplicmonr


30 uo~ssnss!p Zun~oddns rr! 1003 ~ueuodur! u-e s! W!S qo~d .as!p pue suyo~jo ruIea1 art3 puoXsq saAom uoysnssu! Ipun msm woplas suoFssn3srp ysns

-maria Jeqloj palelax uorspap awos Surlleus u! PJOM ISET xo Lluo ayl xa'iau SF 3uaAa auros jo kq!qeqo~d ayl 'sp.rO,?n. Jaylo u1 -auass ayl uo 3<uaXalM sSa3yjo y3eiq Xy*\l, rr~~dxa ley] sa!Joaqx h~ydsuos liue jo L!pq!suas aql pue srasgo ayl jo Xuow!lsal luanbasqns arp arz ~ml~odur; axom -uo~~cw~oju! jo sayd IueAaIas a71 jo auo lilrro s! 1na.s aq~ jo 1b!l~9'e401d ay3 'asmosfo 'pny '93~19 aJaM smsgjo 8u!puodsa~ aql 30 auou zerp asucys lsn[ sea 11 JaylayM cuor3ew.Iojrr! l!atji uo pasaq <Su!p!sap jo ruayqo~d atis ssnss!p euaprus 'qef aql JO pua a43 IQ

.sawosaq faporu lerp suo~.j szp ayl a~!lem.roju! slow a111 clq~~~~ JE~II~!IJE~ e jo ayur!l ayljo axe .Cay] aleme alour ayl Gstro~durnsse 4~1!Ljilf!~dllr~s ~OAE fCm3 Xay t191101.p IE~I aztiza.1 01 auros sluapnls kro!~ct.tz!s 1a83ei at11 01 1apow E %KI!.I~~ILIO~~O ssasold s!q$ q8no.q~ -lapour pagydurfs 3!ayl Xq pa3npoauy sasc!q jo vopsarfp arp $s!pa~d uayo rres Xayl 'sjrp oy ~;ueo Xaxp j~ -(slid asex-awes u! auass arp uo ague axjod aneq <.%-a) zo~s-ej leuo!~ppe ua mnomz o3rir aym 01 lapow aql rasp uc3 Xay~ sases aulos uj .uo!zans!s Tea1 at11 13ajjE suo!lt.lap!suos sno?a.e,z ~orl PIIBJ! apap ~SRW

smapn$s 's1jnsa.i rro!l.elnur!s 3q3 30 anph anys!pald aql au!urJauy 01 JayJo 111 .S~A~OI\U! %~!lapoxu 30 ssaso~d aqs I~C~M prrurrslapun 01 u$aq sxuapn3s xcql ppout Xue o~u! pa~e-tod~osrr! suo!~du~nssc ayl Bu!ssarrppe qSno.rrp st 21 .sl[nsaa .rfayl jo suorlt.s!~druy arp ssnxp pue srro!~drunsse nzau awyl rpp JrraisIsuoa spgoru rrnJ pue pl!nq Xarfi 'ues Xatp 21 arues arp jo aq 01

pual SJ~UIJ~ j! r~o!~sanb II! L~!I!qcqo~d ay3 ~sajje 1y8!w SJ!E~ u! a~!jre usijo sraq.~o aqod 3erp 13Ej ayi ~oq J3p!SU03 01 La~dtu~:~a ~9$ 'pap a."" s~uaprls~ .apzxu ale suo!~druuss~ sno!.rxh '%u!p~!nq rapom jo ssalo~d ayl u~


tl~is type, because it cat1 be used to model engaging problems too complex for students to tackle fortnally. Prob Sirn's simplicity, and dze speed with which suff~cient data can be generated, gives students the time to design a sampling model, collect adequate data, draw conc~usions, and discuss implications of the Gndings. In rhe next section, I show Izotv we build on ideas introduced in modeling probabilistic situations when we move on to data analysis.

3. Data analysis usir~g Datascope

We designed a data-analysis program called "DaraScope" for use in introductofy courses stressing exploratory data analysis in which studerlts work with real data. Our primary objective is for students to learn basic data-analysis techniques for exploring relationships anlong various variables. By using mulriple-variable data sets, we give students the opporcr~tlity to explore questions of particular interest to them. DataScope encourages students to make initial judgments of relationship by visually comparing plots. This is demonstrated below using data obrained from a tluestiorlrtaire administered to 84 students in two high schools in western Massachusetts: Amherst Regional and Holyolce High. Andlerst is a snxall coilege rown, while )-IolyoIre is a larger, industrial city. The i~lfornzatioil collected on tach studenr included !gender, age, birth order, family size, inaritai status of parents, religious activity, rating of school ~erformnnce, educational level of parents, curfew times, working hours and wages, and time spent on homeworlc. Students spend about two weeks exploring various questions irt this data set, among them thc question of whether holding an &r-school job adversely affects school perfornimce.

DataScope encourages exploration by allowing students to form subgroups of some variable based on the values of some other, related, variable. For example, Fig. 4 shows the box plots for hours of homework (HWHRS) and hours spenr working (JOBHRS) for Amherst and Hoiyokce students (in the case of JOBHRS at Amherst, the median is the same as the 1st quartite, as indicated by the double-thick line). The nulnber of cases in each box plot is displayed to the right of the plot. This is imporrant to irrclude as students wiIl frequently draw concfusio~is without considering the number of cases represented by each box plot. The plots below suggest that Amherst students spend more time on homeworlc than working apart-time job, and that the opposite is true of Holyoke students.

T%is tmnd is consistent ryjtl, common (v-6 efdsrereo qpes oLche cawrrs. l'c is tempci11g to ailclude from fig. 4 &at those wit& jobs spend less

time on homework than those without jobs. However, whetl HWHRS are


"grouped" by the categorical variable JOB, students discover the reverse appears to be true - students with a job studied an average of three hours- per-week mare thm those without a job.

HWHRS, AMHERST n=3 1 <+ .".s\ a. -6. j! *@%$d+gq

HWHRS, HOLYOKE n=S1 +=I- +

JOBHRS, WOLYOKE n=5 1 e

Hours per week

Figure 4. Box plots oJbonzemrk. hocm nn~f job hoztrsfir Amhcrst md FIo~okc snldents

job = no ;. **,A .U>. ' c % q * " a < f , . i

n=26 -/f#&4&.7p@&3$$*iq *

job = yes n=56 . .

Homework I-~ours per week

Figure 5. Box plots of honzetuor.4 hottrr fir strrdmt wrri~ am1 zuzrl~wt jobs

At this point, students reconsider their expectarions and develop theories that might explain these data. One possibility that those who worlc also study Inore because they have learned to eFfective1y manage their rime. Or maybe the students with jobs are alder and have more homework assigned. Sorrle of these explanations can be ir~ve-sti~aced by Looking at other variabfes


in the data set. However, among the explanations to consider is the possibility that rhe difference was due to the "Xuck of tile draw" - chat with a bet- ter or larger sample, no difference \vould be found. In my cxpcriencc, stla- dents do not spontaneously raise this possibility, even when they have been using resanipling techniques, as described above to estimate probabilities. 'I'hey will judge differelices between two medians as important in one case, and uniniporralit in another, apparently rnalcing the judg~nent based on the distance between the two medians, as ir appears on the computer screen. Thcy do not sponraneousty evaluate tliu difference with respect to rhe variability (i.e., to the IQI3s as shown in thc box plots). TIrerefoi.e, before showing them how we might determine the of a difference occurriilg by chance, I typically must remirld them that rhis is a possibility.

3.1 Rdn~i'omixatz'on m t s in DataScope

Below is a demonsrration of how Datascope is used ro estimate the one- tailed probabiliry of observing a differellce at least as large as the one observed in the sample above. The method is based on the rantfon~k~tion procedure as origir~ally developed by Fisher (see Barbelfa ct n l , 1390). This involves randomly reordering one of the variables (without replacement) to estimate the probabiliry of the observed difference under the 1lul1 hypothesis. With Datascope, the studenr can first do this "manually", to develop a sense of what the computer is doing. A "reorder" con~rnand randomly reorders rhe values of one of the selccted variables (in chis case, the values for I-IWHRS). That is, d ~ e values of HWHKS are ra~ldornly reassigned to cases. Once reassigned, rbe variable name appears in the data table with the synnbol O 011 both sides to remind the student that the column has been randomized (another command will restore the original order). A box plot of this randomly ordered variable, again grouped by the job variable, can be viewed. Given that the values of HWHRS have been randomly assigned, any difference between the medians of the two groups (with or without jobs? is due ro chance. In the example shown in Fig. 6, this difference is -1 (sub-tracting the tnedian of the tipper plot from the median of the lower

Once students undersrand the procedure, the computer can be instructed to repeatedly reorder the variable and compure the difference between medians of the job and no-job groups, recording these in a new data table as iilusrrated in Fig. 7. lin this case the computer has been instructed to draw 100 random samples. The first few differences that were obtained are shown in Fig. 7 in the "Resampling" data table. The first value is the observed difference, -3. In the background you can see the variables


selected (Vl and GI) in the primaly data rable. The values i t1 HWHRS are bcing randomly reordered.

job = no n=26 -2":-*t',5Pti. ") W$$!FI X +-f------

job = yes n=56 '

$ * x d"" SF&+'aa*? ---*ggJ b

I 1 I 1 I 1

0 5 10 15 20 2 5

OWHESO

Figrrre 6. Horrrrzuo~k hozir~ mndom<~ nfr&t~ra' m job nnB no job growpr

HS Survey 9D

Figure 7. 7irble oj+di~etvncrs Itenuecn medidn IWNI?$fir job (nzd ~zo+b gror<pps lrn stlccefrive resamplig mns.

After [he 100 r a ~ ~ d o m differences have been obtained, the results can be displayed in a histogram as shown in Fig. 8. In this instance, 25 of the samples had a difference at Ieasr as large as -3. Thus, an estimate of idle one- taiied p value is .25. Additional repetitions could be cond~tcred for a more precise estimate. This same procedure is used in DaraScope to test the


statistical significance of a value of I; and of frequency counts i n a 2 X 2 table.

Difference between medians Figtrre 8. Satfipling I~i~rogram dzspliyig rrm/t.r of IOO resdmpIii?t+s

4. Educational outcomes

I have found, as have Simon and Bruce (1991), that students are enthusiasric a b o ~ ~ t roba ability and statistical infererlce when approached througl~ resampling. Bur, do studenrs using this approach learn more than they do in a traditional course! Simon ft.ul. (1976) compared student performance in cotlrses taught t~sing resan~~ling vs. conventional methods. Give11 problems that could be solved using methods taught in either course, students in courses using the resampling approach consistentiy outscored students using the traditional approach.

Many or most students who rake an introductory course will never need to collduct a statistical test or dererminc a probability precisely. They do, however, as members of a complex and increasiligly technological society, require a basic ur~derstaridin~ of uncertainty and the savvy to cvaluare "research" claims in the mass media. Accordingly, tllough sttldents should be able to solve problenls using methods they Iiave been taught, it is even inore important that they understand basic concepts .rvhich underlie these methods. Konofd and Garfield (1993) have developed items ro assess understanding of these basic ideas. Below is orle of our problems, adapted from an itern in Falk (1933, p. 11 1).

The Springfield Meteorologic~l Cenrer wanted to derer~nine the accuracy of their weather forecasts. They searched their rccords for those days when the forecaster had teporred a 70% chance of rain. They compared these faiecasrs ro record? ofwhether or not it actually rained on rhose particular days.


cast of 70% chance of rain can be considered t'ety accurate if i t rained

blem as .I0 this means

ozttToPize of a single trial, I have referred ro this as the "outcome approach" (Konolcf, 1989).

Figure 9. Frequency of reepomes 6&re iiztiuctiopt of193 sdent f ro the zo~atLvpro6lem


21 0 UNDI?RSliV.iDlKG PROBADILITYAND STATISTICS T1iROUCI-I RES&fPLINC

One of my instructional objectives, therefore, is for students to realize that a probability value typically tells us little or notlling about resulrs in the short run, but a great deal about results in the long run. Fig. 10 compares the results on this same problem before (black) and after (gray) instructioit. Correct responses increased only 6% with instruction. The results are similar across the majority of our assessment items. I.it a deeper level, many students after instruction using resampfillg appear tlnaware of the fundamental nature of probability and data analysis.

Selected range

Figitre 10. Frequency ofrefponses befjrc (black) arzd njer CYrny) insirz~ction o f f 99 students to t lx wrn~berpro6lcnz.

formalisms. Perhaps lookirig at development over a singIe course is too srrtall a unit of analysis in the case of and we sltoutd be rhinliing about series of courses over which we can expect to effect and observe conceptual change.


Bibliography

Barbella P., Denby L. and Landwehr J.M. (1990), Beyond exploratory data analysis: The randoznization test, Mathemtics Teacher, 83, pp. 144-149.

Cobb G.W. (1992), Teaching statistics: More data, less lecturing, in L. Steen (ed.), Needing the Callfor Chunge, (MMA Notes # 22, pp. 3-43], Matl~ematical Association of America.

Faik R. (1993), Understanding probabitity and st~~tistics: Problems involving J;k??&erztal concepts, Wellesley, MA.: AK Pc ters (to appear).

Falk R, and Konold C. (1392), The psychology of learning probability, in F.S. Gordon and S.P. Gordon (&cis.), Stztisticsfir the tweudq-first ccntwy (MMA Notes .# 26, pp. 1 5 1- J 64), Mathematicid Associz~ion of America.

Konold C. (1993), Teaching probability t l ~ r o ~ l ~ h rnoddi~lg real proble~ns, Mrzthernrtticr Teacher (to appear).

Konold C. (1969), Informal conceptions of probability, Copition and Instniction, 6 , pp. 59-98.

KonoId C. and Garfield 1. (1 993), StatistiCdl Reasoning Assess7nent. Part I : I~ztuilive 7%izhjng, Scientific Reasoning Research Instittite, University of Massachusetts, Amherst.

Scheaffer 11. L,. (1930), Toward a more quantitatively literate citizenry, The A~lilerican Stfitistichn, 44(1), pp. 2-3.

Simon J.L. and Bruce P. (1991), Resampling: h tool for everyday statistical work, Chance: Nezu Directio~issfor Statistics and Computirzg, 4(1), pp. 22- 32.

Simon J.T., Atkinson D.T. and Shcvokas C. (1976), Prol.ahitity and statistics: Experirnenral results of a radically different teacil~ng method, Anzcrica?~ Mat/~einmticalMonthly, 83, pp. 733-739.

Watkins A., Burril G., Landwehr J. and Schaeffer R. (1992), Rernedial Sratistics?: Thc implications for colleges of the changing secondary scl~ool curriculum, in F.S. Gordon and S.P. Gordon (eds.), Strltisticsjir the twenty-first certtury ( M M A Notes # 26, pp 45-55), Mathematical Association of America.

Note

The curricuium materials and research described in this articie were supported by a grant from the National Science Foundation (Grant #MDR- 8954626). The opinions expressed here are the author's and not necessarily those of the Foundation.


understanding probability and statistical inference through resampling

Documents