the quarterly journal of economics - youth guidance · 2018. 3. 26. · the quarterly journal of...
Post on 14-Feb-2021
3 Views
Preview:
TRANSCRIPT
-
THE
QUARTERLY JOURNALOF ECONOMICS
Vol. 132 February 2017 Issue 1
THINKING, FAST AND SLOW? SOME FIELD EXPERIMENTSTO REDUCE CRIME AND DROPOUT IN CHICAGO*
SARA B. HELLERANUJ K. SHAH
JONATHAN GURYANJENS LUDWIG
SENDHIL MULLAINATHANHAROLD A. POLLACK
We present the results of three large-scale randomized controlled trials(RCTs) carried out in Chicago, testing interventions to reduce crime and dropoutby changing the decision making of economically disadvantaged youth. We studya program called Becoming a Man (BAM), developed by the nonprofit Youth Guid-ance, in two RCTs implemented in 2009–2010 and 2013–2015. In the two studies
∗The title is, of course, a reference to the 2011 book by Daniel Kahne-man, Thinking, Fast and Slow. This project was supported by the Universityof Chicago’s Office of the Provost, Center for Health Administration Studies,and School of Social Service Administration, the city of Chicago, the ChicagoPublic Schools, the Illinois Criminal Justice Information Authority, the EuniceKennedy Shriver National Institute of Child Health and Human Development ofthe National Institutes of Health (awards R21-HD061757 and P01-HD076816),CDC grant 5U01CE001949-02 to the University of Chicago Center for Youth Vi-olence Prevention, grant number 2012-JU-FX-0019 from the Office of JuvenileJustice and Delinquency Prevention, Office of Justice Programs, U.S. Depart-ment of Justice, and grants from the Laura and John Arnold Foundation, theChicago Community Trust, the Edna McConnell Clark Foundation, the CrownFamily, the Exelon corporation, the Joyce Foundation, J-PAL, the Reva and DavidLogan Foundation, the John D. and Catherine T. MacArthur Foundation, theMcCormick Foundation, the Polk Bros Foundation, the Smith Richardson Foun-dation, the Spencer Foundation, the University of Chicago Women’s Board, a pre-doctoral fellowship to Heller from the U.S. Department of Education’s Institute forEducation Sciences, and visiting scholar awards to Ludwig from the Russell Sage
C© The Author(s) 2016. Published by Oxford University Press on behalf of the Presi-dent and Fellows of Harvard College. All rights reserved. For Permissions, please email:journals.permissions@oup.comThe Quarterly Journal of Economics (2017), 1–54. doi:10.1093/qje/qjw033.Advance Access publication on October 10, 2016.
1
mailto:journals.permissions@oup.com
-
2 QUARTERLY JOURNAL OF ECONOMICS
participation in the program reduced total arrests during the intervention periodby 28–35%, reduced violent-crime arrests by 45–50%, improved school engage-ment, and in the first study where we have follow-up data, increased graduationrates by 12–19%. The third RCT tested a program with partially overlapping com-ponents carried out in the Cook County Juvenile Temporary Detention Center(JTDC), which reduced readmission rates to the facility by 21%. These large be-havioral responses combined with modest program costs imply benefit-cost ratiosfor these interventions from 5-to-1 up to 30-to-1 or more. Our data on mecha-nisms are not ideal, but we find no positive evidence that these effects are due tochanges in emotional intelligence or social skills, self-control or “grit,” or a genericmentoring effect. We find suggestive support for the hypothesis that the programswork by helping youth slow down and reflect on whether their automatic thoughtsand behaviors are well suited to the situation they are in, or whether the situa-tion could be construed differently. JEL Codes: C91, C93, D03, D1, I24, I3, I32,K42.
I. INTRODUCTION
Disparities in youth outcomes in the United States are strik-ing. For example, for 15–24-year-olds, the male homicide rate in
Foundation and LIEPP at Sciences Po. For making this work possible we are grate-ful to the staff of Youth Guidance, World Sport Chicago, the Chicago Public Schools,and the Cook County Juvenile Temporary Detention Center, and to Ellen Al-berding, Roseanna Ander, Mayor Richard M. Daley, Anthony Ramirez-DiVittorio,Earl Dunlap, Mayor Rahm Emanuel, Wendy Fine, Hon. Curtis Heaston, MichelleMorrison, Dave Roush, and Robert Tracy. For helpful comments we thank LarryKatz, Andrei Shleifer, four anonymous referees, Stefano DellaVigna, Dan Gilbert,John Rickford, and seminar participants at Case Western, Columbia, Duke, Eras-mus, Harvard, MDRC, Notre Dame, Northwestern, Sciences Po, Stanford, Univer-sity of Chicago, University of Miami, University of Michigan, University of Penn-sylvania, University of Toronto, University of Virginia, University of Wisconsin,Yale, the MacArthur Foundation, NBER, New York City Department of Proba-tion, and the joint New York Federal Reserve/NYU education workshop. For helpaccessing administrative data we thank the Chicago Public Schools, the ChicagoPolice Department, and ICJIA, for providing Illinois Criminal History Record In-formation through an agreement with the Illinois State Police. For invaluable helpwith monitoring, data collection, and analysis we thank Nour Abdul-Razzak, SamCanas, Brice Cooke, Stephen Coussens, Gretchen Cusick, Jonathan Davis, NathanHess, Anindya Kundu, Heather Sophia Lee, Duff Morton, Kyle Pratt, Julia Quinn,Kelsey Reid, Catherine Schwarz, David Showalter, Maitreyi Sistla, Matthew Veld-man, Robert Webber, David Welgus, John Wolf, and Sabrina Yusuf. The findingsand opinions expressed here are those of the authors and do not necessarily reflectthose of the Department of Justice, National Institutes of Health, the Centers forDisease Control, or any other funder or data provider.
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 3
2013 was 18 times higher for blacks than whites (71 versus 4per 100,000).1 Black males lose more years of potential life be-fore age 65 to homicide than to America’s leading overall killer—heart disease.2 A large body of research emphasizes that—beyondinstitutional factors—choices and behavior also contribute tothese outcomes, including decisions around dropping out of highschool, involvement with drugs or gangs, or responses to con-frontations that could escalate to serious violence. In this articlewe present the results of three large-scale randomized controlledtrials (RCTs) that seek to reduce crime and dropout by changingthe decision making of disadvantaged youth at elevated risk forthese outcomes.
Given the current focus of U.S. social policies aimed at chang-ing these behaviors, one noteworthy feature of the interventionswe study is what they are not. They do not involve early childhoodeducation, or academic skill development, or vocational or techni-cal training, or subsidized jobs or internships, or cash transfers, orin-kind supports, or efforts to change parenting or the home envi-ronment, or any sort of incentive scheme for children, parents, orteachers. Another noteworthy feature is that, unlike most previ-ous interventions for low-income youth, those we study here seemto generate large impacts on important behavioral outcomes.
To provide some concrete sense of what the programs are,we describe the first activity youth do in one of the programswe study: the Becoming a Man (BAM) program developed by theChicago nonprofit Youth Guidance (YG). Students are divided intopairs and one is given a ball, which the other student is told he has30 seconds to get from his partner. Almost all youth use physicalforce to try to take the ball out of the other’s fist. During thedebrief, the group leader points out that no one simply asked forthe ball. When prompted about why they did not simply ask, mostrespond with some version of “he wouldn’t have given it,” or “hewould have thought I was a punk.” The leader then asks the otheryouth, “How would you have reacted if asked nicely for the ball?”The answer typically is something like, “I would have given it;it’s just a stupid ball.” This exercise, like many in the program,teaches youth to think more carefully about the situations theyare in.
1. Our calculations compare non-Hispanic blacks to whites and focuson homicides (excluding fatalities from legal intervention); see WISQARS(http://webappa.cdc.gov/sasweb/ncipc/mortrate10 us.html).
2. WISQARS (http://webappa.cdc.gov/sasweb/ncipc/ypll10.html).
http://webappa.cdc.gov/sasweb/ncipc/mortrate10_us.htmlhttp://webappa.cdc.gov/sasweb/ncipc/ypll10.html
-
4 QUARTERLY JOURNAL OF ECONOMICS
The interventions we study here were carried out in very dis-advantaged neighborhoods on the south and west sides of Chicago.Our first two RCTs tested BAM, first randomizing 2,740 youth to aone-year program in the 2009–2010 academic year (AY), with thesecond RCT stretching the curriculum out over two years in AYs2013–2014 and 2014–2015 with 2,064 youth.3 We measure out-comes with longitudinal government administrative data. In bothstudies the effect on program participants during the programperiod were similar, reducing total arrests by 28–35%, violent-crime arrests by 45–50%, and arrests for other crimes by 37–43%.While these impacts on arrests did not persist beyond the pro-gram period in the first BAM study (we do not have postprogramdata for the second BAM study), we did find persistent impactson schooling outcomes—including gains in high school graduationrates of 6 to 9 percentage points (12–19%).
Our third RCT was carried out in a very different setting—the Cook County, Illinois Juvenile Temporary Detention Center(JTDC), where high-risk juvenile arrestees in the Chicago areaare taken for pretrial detention. The intervention, carried out insome residential units in the JTDC but not others, consisted ofa package of reforms that included a token economy for good be-havior inside the facility, increased educational requirements forstaff, and a daily program delivered by the detention center’s staffthat had many elements similar in spirit to BAM. We focus on the2,693 male admissions to the JTDC from 2009 to 2011 who wererandomly assigned to units with or without the reforms, and forwhom we have at least 18 months of follow-up data. Receipt ofprogramming reduced readmission rates by 16 percentage points(21%) and had impacts on the number of readmissions or read-missions plus arrests that were similarly large but sometimes lessprecisely estimated.
What is striking about these interventions is not just thatthey generated such large behavioral responses but also that theywere able to do so at relatively low cost—less than $2,000 perparticipant (and sometimes much less). In our first BAM RCT,for which we have the most complete data on outcomes, we esti-mate that the value of crime reduction alone yields benefit-cost
3. The program is also abbreviated as “B.A.M.” but for consistency we use acommon style for all acronyms. Some youth also received after-school programmingdeveloped to reinforce the BAM curriculum by a nonprofit called World SportChicago. We argue below that our results are not due simply to incapacitation ofyouth during after-school hours when sports were held nor to sports more generally.
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 5
ratios that range from 5-to-1 up to 30-to-1. These are likely tobe lower-bound estimates, given our findings that BAM increaseshigh school graduation rates as well.
Why do these programs work so well? Existing theories ofwhat (beyond academic skills) determines people’s life outcomessuggest a few possible channels such as self-control, conscien-tiousness, “grit,” emotional intelligence, social skills, support frompro-social adults (“social capital”), or an understanding of the re-turns to education. To measure these candidate mechanisms wehave surveys administered throughout the Chicago Public School(CPS) system, including to youth in our first BAM study. Thesedata are not totally ideal, partly because they captured responsesfrom just under half of the youth in our study and partly becausethey did not capture measures related to every possibly relevanttheory from the prior literature. With these caveats in mind, wefind no positive evidence that any of the candidate mechanismssuggested by prior research for which we have measures explainmuch of the treatment effect. Our estimates of treatment effectson these potential mediators, and the relationship between media-tors and outcomes, suggest that these mechanisms are unlikely toaccount for more than a modest share of the program effects on be-havioral outcomes. Using arrest records with exact date of arrest,we can also rule out that the effects were due merely to keepingyouth busy (voluntary incapacitation) on days when after-schoolactivities occurred.
When we look at the curricula for the programs we studyhere, we can see why commonly discussed mechanisms do notseem so important for these interventions. Consider, for example,BAM’s “fist” exercise described above. This does not seem to act oncommonly discussed mechanisms like self-control or social skillsor grit. The exercise does not help youth learn how to either rec-ognize or stifle feelings of anger, or learn the most polite way torequest the ball, or how to persist in trying to get the ball. Manyother BAM activities are similar in that regard.
Another noteworthy aspect of the fist exercise (and the BAMprogram as a whole) is that it does not tell youth the “right”thing to do. BAM providers recognize that these youth live indistressed neighborhoods where being aggressive or even fight-ing may—unfortunately—sometimes be necessary to avoid devel-oping a reputation as someone who is an easy victim. It is nothard to see how someone navigating that sort of neighborhoodenvironment could develop a tendency to reflexively push back
-
6 QUARTERLY JOURNAL OF ECONOMICS
against being challenged. That response can lead to trouble if it isovergeneralized and sometimes applied in settings where it is nothelpful—such as school.
This illustrates our own hypothesis for why these programschange youth behavior, based on the psychology of automaticity.4
Because conscious deliberation is mentally costly, all of us de-velop a series of automatic responses that are usually adaptive tosituations that we commonly face—such as youth in distressedurban neighborhoods fighting back aggressively when challenged.Why might a teenager overgeneralize and deploy this response ina setting where it is not adaptive, such as in school? Behavioral sci-ence shows that we all make automatic assumptions about whatsituation we are in and that these assumptions can sometimes bewrong (Ross and Nisbett 2011). Being yelled at by a teacher inschool to stop talking so class can begin may at first glance feellike one’s reputation is being challenged, just as on the street. Wehypothesize that these interventions improve youth outcomes insubstantial part because they help youth slow down in high-stakessettings, examine their automatic assumptions about what situa-tion they are in, and ask whether the situation could be construeddifferently. That is, the programs help youth have a greater senseof occasion.5
The difference between this theory and other commonly dis-cussed mechanisms is highlighted by the fact that the BAMproviders say some of their youth have the problem of not be-ing aggressive enough. That is, rather than having the problem oftaking the automatic response adaptive for the neighborhood andovergeneralizing to the school setting, some youth have the op-posite problem of overgeneralizing the response that is adaptivefor school—which in turn gets them into trouble in the neighbor-hood. For these youth, BAM can have the effect of increasing thefrequency with which they assert themselves—ideally only in theright situations.
To test this theory, we administered a decision-making exer-cise (an iterated dictator game) to about half of the youth in oursecond BAM study. This exercise made youth think they had beenprovoked by a classmate and then gave them a chance to retaliate.
4. Within psychology automaticity is most notably associated with dual sys-tems models, which are well summarized in Kahneman (2011). Other economicmodels of dual systems thinking include Cunningham (2015) and for impulse con-trol Fudenberg and Levine (2006).
5. We thank our colleague Dan Gilbert for suggesting this wonderful phrase.
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 7
Our theory predicts BAM youth should slow down and spend moretime thinking about how to respond. Our theory does not predicthow they will respond, since that depends on what situation theyconstrue themselves to be in. Consistent with our theory, BAMincreased decision-making time in response to the provocation by80%. In terms of the amount of retaliation administered, we foundfew differences between BAM youth and controls, which does notseem consistent with mechanisms that emphasize changes in fac-tors that would make youth uniformly more pro-social across allsituations.
This hypothesis gives us a way to understand why these inter-ventions are more successful than so many previous efforts. Manysocial policies intended to help disadvantaged youth try to changebehavior by changing the long-term returns to pro- or antisocialbehavior, usually with disappointing results. More promising maybe to directly help youth recognize their automatic assumptionsand responses and make better decisions in high-stakes situa-tions. As one JTDC staff member told us, “20% of our residentsare criminals; they will harm other people if they are not lockedup. But the other 80%, I always tell them—if I could give you backjust 10 minutes of your lives, you wouldn’t be here.”6
II. INTERVENTION STRATEGY
This section describes the two interventions we study, keyparts of which may be lumped together under the broad heading ofwhat psychologists call cognitive-behavioral therapy (CBT). CBTis designed to get youth to “think about their thinking,” or engagein “meta-cognition” (Beck 2011).7 There are other elements of the
6. Personal communication, Darrien McKinney to Jens Ludwig, Sendhil Mul-lainathan, and Anuj Shah, October 18, 2012.
7. CBT programs vary in their focus, including the degree to which they try toreduce automaticity, and not all interventions to reduce automaticity will neces-sarily be called CBT. Since the 1970s, CBT has been used to address mental healthdisorders such as substance abuse, anxiety, and depression and can be more ef-fective than antidepressant drugs (Rush et al. 1977). Since then, there has beengrowing practitioner interest in using different versions of CBT as a social policytool to address socially costly behaviors. Yet there is little good evidence currentlyabout effects on those behaviors of greatest policy concern such as youth delin-quency, violence, and dropout. (This is discussed in detail in the Online Appendix,specifically see Online Appendix A and Tables A.1 and A.2). One recent excep-tion is Blattman, Jamison and Sheridan (forthcoming) who found a CBT programfor adults in Liberia was successful, especially when combined with cash grants.
file:qje.oxfordjournals.orgfile:qje.oxfordjournals.org
-
8 QUARTERLY JOURNAL OF ECONOMICS
two programs we study that only partially overlap. This providesus with one source of leverage for learning more about underlyingmechanisms.
II.A. Becoming a Man (BAM)
The BAM program was developed by Youth Guidance abouta decade before our first RCT of the program in AY 2009–2010.The program was operating in a single Chicago high school anda few elementary schools before being taken to scale for our RCT.In the first experiment, BAM offered youth the opportunity toparticipate in 27 one-hour, once-a-week group sessions held dur-ing the school day over a single school year. The intervention isdelivered in groups, which helps control costs, with groups keptsmall (assigned groups of no more than 15 students and averagerealized groups of about 8) to help develop relationships. Studentsskip a class to participate, which is a draw for some youth. In ourfirst BAM RCT some youth were also offered the chance to par-ticipate in after-school sports programming delivered by WorldSport Chicago, to increase participation and (because coacheswere trained in parts of the BAM curriculum) to reinforce theprogram.8 The second BAM RCT was carried out in AYs 2013–2014 and 2014–2015, with the curriculum stretched out over twoyears (up to 45 sessions) so providers could go into more depth oneach topic as well as cover more advanced material (particularlyrelated to self-reflection and skill building). In this second studysports were also offered in the first year but very little in thesecond year, which helps us isolate the effect of BAM from sports.
Table I illustrates a few of the key types of activities includedin the BAM curriculum and provides a brief description of each se-lected activity. The program is manualized and can be delivered bycollege-educated men without specialized training in psychologyor social work, although YG had a preference for such training inselecting program providers. YG also prioritized hiring counselorswho were able to keep youth engaged, and often hired people fromneighborhoods similar to those in which they would be working.
The curriculum includes standard elements of CBT (Beck2011), such as a common structure to most sessions that starts
That program had a variety of behavioral components, including teaching angermanagement and self-discipline.
8. The sessions were one to two hours each and included nontraditional sportslike archery, boxing, wrestling, weightlifting, handball, and martial arts.
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 9
TABLE ISELECT BAM ACTIVITIES
Activitycategory Example activities
Immersive/experiential
The Fist: Students are told to get an object from apartner. Many try to use force. The counselor asksquestions to highlight how their partners were willingto give up the object if they calmly requested it.Plates: Students reflect on what it has taken tosuccessfully complete group missions and write thoseattributes on a plate. The plates are placed on the floor,and students must cross the floor by using the plates.However, if no one is standing on a plate, then it isremoved (making the task more difficult).Trust Walk: Students follow group leaders around theschool silently and without disrupting the school. Theyare told that with freedom comes responsibility.Focus Mitt Drill: Students punch focus mitts for anextended period.Human Knot: Students stand in a circle and grab thehands of someone standing across from them. Theymust then untangle themselves without letting go.
Reflective/introspective
Check-ins: Students talk to each other about whatthey are doing well and areas where they still need toimprove. Students must listen patiently while someoneelse discusses their attributes.
Role-playing Our Story of What Happened: Students imagine aconflict and discuss why the conflict came about. Theyexamine thinking distortions that might have madethe conflict worse.High School Day: Students do a role-play where astudent and administrator have a confrontation. Theyact out the conflict with “out of control” and “in control”anger expressions.$10 Role-play: Students role play a student borrowingmoney and then never paying it back.
Skill-building Cognitive Thought Replacement: Students learnhow to recognize negative thoughts that arise and howto replace them. It is not necessary to replace negativethoughts with positive thoughts, but rather to insteadfocus on what can be done to control the situation thatis leading to the negative thought.Manhood Questions and Rites of Passage:Students discuss the key moments when boys becomemen and various rites of passage that exist.Positive Anger Expression: Students are taughtabout how to express anger in a controlled way.
-
10 QUARTERLY JOURNAL OF ECONOMICS
TABLE I(CONTINUED)
Activitycategory Example activities
Stories anddiscussion
Rudy: Students watch and discuss the movie Rudy.Before beginning the movie, the counselor holds up $2and asks who wants the money. Even as students raisetheir hand, he keeps asking who wants it untilsomeone simply takes it from him. He explains that weoften overlook opportunities, but the student who tookthe money saw it as an opportunity and took a chance.The Boy Who Cried Wolf: Students listen to anddiscuss the story where one day a boy pretends that heis being attacked by a wolf. He is amused by how histown responds to this prank. So when he feels bored onanother day, he does it again. And again. He promisesto stop playing around, but when he feels bored hecan’t help but do it again. In the end, when he isactually attacked by a wolf, no one responds to hispleas for help.Miracle: Students watch and discuss the film Miracleabout the U.S. men’s hockey team.
with a “check-in.” Youth sit in a circle with the counselor, who re-flects on how things in his life are going in various domains. Theyouth then follow suit. This is an example of what we call “retro-spective/introspective” activities, which include various efforts toget youth to talk about the things they are doing well and areas inwhich they still need to still improve, and also share what othersare doing well and need to improve.
Another type of activity in the BAM curriculum we call “im-mersive or experiential,” of which the fist exercise described aboveis one example. Another example is called the stick. Youth aredivided into two groups and lined up facing each other. They aretold to put their arms out chest high and extend their index fin-gers, and the counselor then lays a 10- or 15-foot plastic pipeacross everyone’s fingers. The group is then told that they mustlower the pipe to the floor but their fingers must be touching thepipe at all times. This leads everyone to put upward pressure onthe pipe, which makes it go up rather than down. As youth becomeimmersed in the activity, they can lose themselves in the momentand become frustrated, blaming each other rather than recogniz-ing that each of them contributes to the problem (and that theycould help solve the problem themselves by trying to coordinateand lead the group).
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 11
Other types of activities included in the BAM curriculum arewhat we call “role-playing” and “stories and discussions.” For ex-ample, in the $10 role-play activity, students act out a scene inwhich one of them has borrowed money from another but thennever paid it back. The youth act out how they would respondand then the group discusses what happened and why, and whatmight have led to a better outcome. Stories include the elephantand the rhino, in which two large animals are very persistent intheir refusal to make way for the other, to both their detriments.
The program also does some “skill building.” This includeslessons in muscle relaxation, deep breathing, and channelinganger productively. It also includes cognitive thought replace-ment, a CBT element that asks youth to identify and replaceproblematic or false beliefs. Finally, the curriculum includes a dis-cussion of different conceptions of masculinity and some generalvalues like the importance of integrity and personal accountabil-ity. It also takes youth on field trips to local colleges to highlightthe value of education, and, by putting youth in regular contactwith a pro-social adult, has a mentoring component as well.
II.B. Juvenile Detention
The setting for our third RCT is the Cook County JuvenileTemporary Detention Center, which is where the highest-risk ju-venile arrestees in Cook County are taken after they are arrested.Youth are held for an average of three to four weeks until theircases are adjudicated in court, although youth whose cases are be-ing heard in the adult court system can be detained much longer.In May 2007 the JTDC began to implement a series of reformsthat included the use of a token economy system to help main-tain order and twice-daily participation in group CBT sessionswhen youth were not attending the school inside the JTDC, re-placing time that had typically been spent watching TV. The CBTprogram used a manualized curriculum9 and was delivered bytrained JTDC staff. Partly to help implement these reforms, theJTDC also required increased educational requirements for staffworking in the newly reformed centers.
9. The specific intervention studied here was developed by Dr. Bernie Glosand his associates from the DuPage County, Illinois Juvenile Detention Center.The curriculum is adapted from the best material from several prior CBT modelsthat had been used in detention and is based in part on the cognitive behavioraltraining ideas from Maultsby (1975, 1990) (see also Ellis 1957; Ellis and Harper1975).
-
12 QUARTERLY JOURNAL OF ECONOMICS
Table II summarizes a few key types of activities and spe-cific activities included in the JTDC curriculum. While BAM uses“check-ins” at the start of most sessions to get youth to engage inreflection or introspection, the JTDC program requires youth tocarry out “thinking reports” every time their misbehavior causesdetention staff to give them a “time out” (a certain amount oftime alone in their cell). Examples of other activities in the reflec-tive/introspective category in the JTDC include retrospectivelytalking through experiences and focusing on what an outside, ob-jective observer would have seen (or, taking a “camera view”). Theprogram in the JTDC also continually emphasizes the importanceof learning to “stop, look, and listen.”
While BAM emphasizes youth engagement and seeks to“show, not tell,” the JTDC program in comparison is much more“tell, not show.” In the JTDC curriculum there are no immer-sive/experiential activities (like the stick or the fist) or even anyphysical activities (including no sports). In addition to reflec-tive/introspective activities, “skill-building” activities are commonin the JTDC curriculum. As in BAM, the “skill-building” curricu-lum focuses on helping youth “keep cool when they are angry”(using anger expression and relaxation techniques), as well as onthings like setting goals, interpersonal problem solving, and pay-ing attention to one’s feelings. The token economy is often used toreinforce the CBT curriculum by rewarding positive behaviors orthoughts consistent with these lessons.
III. EVIDENCE FROM THREE RANDOMIZED CONTROLLED TRIALS
This section presents the results of our three large-scaleRCTs, two of which tested BAM and the third of which testeda related program implemented inside the Cook County JuvenileTemporary Detention Center. The exact outcome measures andtime horizons we examine are not identical across all three stud-ies, but the results are qualitatively consistent in showing sizableyouth responses on different measures of criminal behavior orschooling.
III.A. Studies 1 and 2: Becoming a Man
1. Samples and Randomization. For our first RCT of BAM(hereafter “study 1”), during the summer of 2009 we recruited18 elementary and high schools in the CPS located on Chicago’s
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 13
TABLE IISELECT JTDC ACTIVITIES
Activitycategory Example activities
Reflective/introspective
Self-talk: Students are taught about how the mindalways tries to make sense of what is going on and howthese thoughts drive our behavior. For example, thecounselor might hold out his hand and see how peoplerespond. He then explains how the students’ mindshave an automatic interpretation of and reaction to hisoutstretched hand.Hot Button Situations: Students talk aboutsituations that make them upset. They describe thesituation and their thoughts in that situation. Theyidentify elements of “hot” self-talk that leads tonegative consequences and they identify hot buttonsituations that trigger these thoughts.Camera Check: Students imagine a hot buttonsituation and then describe how they would navigateit. They then imagine the situation again from aneutral outsider perspective.Rational Self-Analysis (RSA): After antisocialbehavior, students complete an RSA, writing down thefacts of the incident, identifying what self-talk/feelingsled to the behavior, reporting what a camera wouldhave seen, and brainstorming alternative/moreadaptive self-talk. Youth then process their RSA withstaff and discuss the new self-talk options they havedeveloped.
Skill-building Goal Setting: Students are encouraged to make oneconcrete statement about something they want to dobetter or differently.Goals and Choices: Students discuss what they wantversus what they need. And they discuss how goals,wants, and needs are always set internally. No one elsecan set them for you. Students talk about “big wins”that they want to achieve and think about how theycan break down long-term goals into shorter, moremanageable pieces to help them achieve their goals.Keeping Cool When You Get Angry: Studentsdiscuss how situations can drive angry self-talk, whichleads to negative outcomes. They are taught aboutvarious cognitive distortions. They then learntechniques to physically calm down and to replacenegative or angry thoughts.
-
14 QUARTERLY JOURNAL OF ECONOMICS
TABLE II(CONTINUED)
Activitycategory Example activities
Skill-building Me Mode and We Mode: Students discuss elementsof self-talk that are focused only on one’s own needsinstead of other people’s needs.Problem Solving: Students are given a 6-stepapproach for solving problems that involves identifyingthe problem, thinking about several solutions, andpicking the best solution.
Stories anddiscussion
Thinking Patterns: Students are shown severaloptical illusions that can be seen in two ways. A lessonfollows about how the mind sometimes only sees oneinterpretation or how it only sees what it expects tosee. Students fill out a sheet on their expectationsabout their lives and basic rules for their lives.Moral Development Groups: Students arepresented with morally ambiguous situations, and areasked to identify various potential outcomes forthemselves and others based on different responses.
Other Drugs and Alcohol: Students use the frameworkthey’ve developed to specifically focus on situationsinvolving drugs and alcohol.
low-income, racially segregated south and west sides, where thecity’s violent crime is disproportionately concentrated (see OnlineAppendix Figure A.1). Our sample was essentially the 2740 7th-to 10th-grade male students at highest risk of failure in theseschools, after excluding students who rarely attended school (andso would not benefit from a school-based intervention) or hadserious disabilities. This sample represented around 75% of allmales in grades 7 through 10 in the study schools (see OnlineAppendix B). Our second BAM RCT (“study 2”) was carried out in2013–15 with 2064 male 9th and 10th graders attending nine CPShigh schools. Similar sample eligibility and randomization proce-dures were used for study 2, though the larger number of BAMslots per school meant study 2 covered a broader risk spectrumthan did study 1.
Both studies were block-randomized experiments, where stu-dents were the unit of randomization and were randomly assignedwithin schools (study 1) or school-by-grade “blocks” (study 2).In study 1 youth were randomized to one of three treatmentarms (in-school BAM, after-school sports programming that
file:qje.oxfordjournals.orgfile:qje.oxfordjournals.orgfile:qje.oxfordjournals.orgfile:qje.oxfordjournals.org
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 15
incorporated BAM elements, or both) or the control group.10 Un-fortunately, due in part to treatment-arm crossover, study 1 doesnot have adequate statistical power to disentangle the separateeffects of BAM from the after-school program; in our main anal-yses, we pool the treatment arms together (results separately bytreatment arm and details about crossover are in Online AppendixC, Figure A.2 and Table A.11). In study 2 youth were randomlyassigned to either be offered BAM for two years (2013–14 and2014–15) or to the control group. There were some sports sessionsoffered in five of the nine schools in study 2, but sports participa-tion was low in the first year and then declined by 80% from thefirst to second year. We return to this below.
Table III shows that both studies enrolled very disadvan-taged samples of youth, and that random assignment appears tohave been carried out correctly. In both studies youth were about15 years old at baseline,11 with one-third to one-half being old forgrade. They missed on average eight weeks of school in study 1(when the school year was 170 days) and six weeks in study 2(when the school year was 180 days); many had been arrestedbefore. Reflecting the composition of their neighborhoods, around70% of youth are black and the remainder Hispanic. In neitherstudy can we reject the null hypothesis that the set of base-line characteristics is the same for treatment and control groups(F(18,2543) = 1.04, p = .409 and F(18,1752) = 0.38, p = .99, re-spectively).12 We also find that the youth in study 1 were moredisadvantaged on average than those in study 2; we can rejectthe null hypothesis that the difference in study 1 versus study 2baseline means are jointly zero, F(18,4348) = 329.31, p < .001.13
10. Three of our 18 schools could not offer after-school programming because oflogistical or space reasons. Eight schools offered both in- and after-school treatmentarms in some combination but not all three treatment arms.
11. Even though study 1 covers youth in grades 7–10 and study 2 coversyouth in high school grades only (9–10), the average age is slightly higher in study1 because of a larger number of youth 17- or 18-years-old at baseline.
12. The baseline variables in the joint test are: age; grade; number of in-schoolsuspensions; number of out-of-school suspensions; number of each type of arrest(violent, property, drug, and other); number of each type of grade earned (A throughF); and indicator variables for being black, Hispanic, old-for-grade, and having alearning disability.
13. The difference is not just because of age or school differences in the sample;when we hold age constant and compare youth in 9th and 10th grade in just thethree schools that are common to both BAM studies, we still see that the study 1youth are more disadvantaged on average than those in study 2.
file:qje.oxfordjournals.org
-
16 QUARTERLY JOURNAL OF ECONOMICS
TABLE IIIBECOMING A MAN STUDIES 1 AND 2: BASELINE CHARACTERISTICS
Study 1 Study 2
Control Treatment Control Treatment
No. of students 1,267 1,473 1,048 1,016
Baseline characteristic
DemographicsAge 15.70 15.51 14.75 14.81Black 72% 69% 70% 68%Hispanic 28% 31% 28% 30%
SchoolingGrade 9.42 9.29 9.41 9.46Old for grade 55% 51% 35% 35%GPA 1.68 1.73 2.11 2.16Days present 129.86 133.60 148.18 149.78Learning disability 20% 19% 17% 16%
CrimeAny baseline arrests 37% 35% 23% 23%
No. of baseline arrests for:Violent offenses 0.35 0.35 0.19 0.18Property offenses 0.21 0.19 0.14 0.13Drug offenses 0.17 0.18 0.11 0.14Other offenses 0.45 0.47 0.29 0.32
p-value on F-test oftreatment-controlcomparison for allbaselinecharacteristics
.409 .991
Notes. Asterisks indicate statistical significance of pairwise treatment-control comparison for a given base-line characteristic controlling for randomization block fixed effects with heteroscedasticity-robust standarderrors. Data from Chicago Public Schools administrative data, Illinois State Police arrest records (study 1),and Chicago Police Department arrest records (study 2). Means calculated using non-missing observationsfor each variable. Pre-program arrests are arrests prior to start of program school year. For study 1, thebaseline school year (AY2009–10) was 170 days; for study 2, the baseline year (AY2013–14) was 180 days.GPA is measured on a 0–4 scale. Joint significance tests for equality of all baseline characteristics use onlynonmissing data (n for joint tests: study 1 = 2,579, study 2 = 1,770). ∗ p < .10, ∗∗ p < .05, ∗∗∗ p < .01.
At least part of the difference in baseline schooling characteristicsacross studies may be due to general improvements in reportedschooling outcomes throughout CPS over time.14
14. For example, reported graduation rates have increased from 57% in 2010–11 to 70% for 2014–15, while the reported CPS 9th grade indicator for being “ontrack” for graduation (Allensworth and Easton 2005) increased from 69% to 84%over that period (see http://cps.edu/SchoolData/Pages/SchoolData.aspx).
http://cps.edu/SchoolData/Pages/SchoolData.aspx
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 17
In both BAM studies about half of those randomized totreatment actually participated (defined as attending at leastone program session). This take-up rate is consistent withmany other social experiments despite the fact that we ran-domized (using administrative data) prior to seeking con-sent for program participation, in contrast to the more com-mon practice of consenting and then randomizing.15 We sus-pect participation rates for the after-school programming instudy 1 are understated because of record keeping issues; OnlineAppendix C and Table A.4 discuss how we handle this issue.Participants attended on average 13 sessions the first year ofstudy 1, and for study 2 an average of 17 and 21 sessions dur-ing the first and second years, respectively (see Online AppendixTable A.3). A small share of controls also received program ser-vices in both studies.
2. Data and Outcome Measures. Our main schooling out-comes come from longitudinal student-level CPS records. We havethese data through AY 2014–15, which for study 1 covers the pro-gram year plus five follow-up years, and for study 2 covers only thetwo program years. We create a summary index of three school-ing outcomes in Z-score form (GPA, days present, and enrollmentstatus at the end of the year), which we call “school engagement.”Use of an index reduces the number of hypothesis tests, which re-duces the risk of false positives (Westfall and Young 1993; Kling,Liebman, and Katz 2007; Anderson 2008), and improves statisti-cal power to detect effects for outcomes within a given “family” ofoutcomes that are expected to move in a similar direction. We im-pute group means for missing outcomes, which assumes data aremissing completely at random. Our results are similar when werelax this assumption and use multiple imputation or other ap-proaches to handling missing data (Online Appendix Table A.10).For study 1, where we have longer-term follow-up data, we arealso able to examine impacts on high school graduation rates.
To measure effects on criminal behavior, for study 1 we useelectronic arrest records (“rap sheets”) from the Illinois State Po-lice (ISP). For study 2 we use arrest data from the Chicago PoliceDepartment (CPD). Both datasets are linked to our samples usingprobabilistic matching on first and last name and date of birth.
15. Consent was for program participation only; outcome data are availablefor all youth who were randomized.
file:qje.oxfordjournals.orgfile:qje.oxfordjournals.orgfile:qje.oxfordjournals.orgfile:qje.oxfordjournals.org
-
18 QUARTERLY JOURNAL OF ECONOMICS
Arrest records avoid the problem of under-reporting of criminalinvolvement in survey data (Kling, Liebman, and Katz 2007) butrequire the assumption that the intervention does not change thechances a crime results in arrest. Because intervention impactscan vary by crime type, we present results separately for violent,property, drug, and “other” crimes.16 We cannot distinguish “miss-ing data” from “no arrests,” so we cannot explore how the arrestimpacts change when we change how we handle missing data.
3. Estimation Approach. Given our randomized experimen-tal design, our analysis plan is quite straightforward. Let Yistdenote some post-program outcome for individual i at school sduring post-randomization period t, which is a function of treat-ment group assignment (Zis) and data from government recordsmeasured at or before baseline (Xis(t-1)) as in equation (1). Wecontrol for baseline characteristics to improve precision by ac-counting for residual variation in the outcomes (results with-out baseline covariates are similar).17 We also control for the“randomization block” with school (study 1) or school-by-grade(study 2) fixed effects (γ s). The intention to treat effect (ITT) cap-tures the effect of being offered the chance to participate in the pro-gram, and is given by the estimate of π1 in equation (1). We presentrobust standard errors but do not cluster by school, partly be-cause the fixed effects account for within-school or within-school-and-grade correlations across students in mean outcomes. As asensitivity analysis we also calculate p-values that come from apermutation test, which randomly reassigns the treatment-offerindicator Zis and does not rely on distributional assumptions orany asymptotic theory,18 as well as p-values that account for mul-tiple comparisons (discussed below).
(1) Yist = Zisπ1 + Xis(t−1)β1 + γ1s + ε1ist.
16. We exclude arrests for motor vehicle violations, but results are similarincluding them (Online Appendix Table A.12).
17. We control for: days present; number of in-school suspensions; number ofout-of-school suspensions; number of each type of grade received (A, B, C, D, F);dummies for ages 14–15, 15–16, and 17+; and indicators for having a learningdisability, being in 9th or 10th grades, being old-for-grade, being black, beingHispanic, and having one, two, or three and over arrests of each type. For missingbaseline covariates, we assign a value of zero and include an indicator that thevariable is missing.
18. See, for example, Young (2015) for how the use of re-randomization testscan matter in practice.
file:qje.oxfordjournals.org
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 19
In addition we present the effects of participating in the pro-gram (defined as having attended � 1 session) for those who par-ticipate, which we estimate using two-stage least squares withrandom assignment (Zis) as an instrumental variable (IV) for par-ticipation (Pist), as in equations (2) and (3) (Bloom 1984; Angrist,Imbens, and Rubin 1996). This assumes treatment assignmenthas no effect on the outcomes of youth who do not participate inthe intervention.
(2) Pist = Zisπ2 + Xis(t−1)β2 + γ2s + ε2ist,
(3) Yist = Pistπ3 + Xis(t−1)β3 + γ3s + ε3ist.
Since a small share of controls gets the program, π3 is tech-nically a local average treatment effect (LATE). But given the lowrate of crossover this should be close to the effect of treatment onthe treated (TOT). We benchmark the size of these effects withthe control complier mean (CCM) (see Katz, Kling, and Liebman2001), but given the treatment crossover we estimate this usingthe formula from Heller et al. (2013). If C indicates being a “com-plier” and Z indicates treatment assignment, we calculate this asCCM = E(Y|C = 1, Z = 1) – [E(Y|C = 1, Z = 1) – E(Y|C = 1,Z = 0)]. The term in brackets is our LATE estimate. However, wemust recover the first right-hand-side term, E(Y|C = 1, Z = 1),since what we observe in the data is the mean outcome for alltreatment group participants—a weighted average of the meanoutcomes for compliers and always-takers. Let P indicate actualparticipation and A be an indicator for always-takers. Then:
E(Y |Z = 1, P = 1) = E(Y |Z = 1, C = 1)(
1 − E(A|Z = 1)E(P|Z = 1)
)(4)
+ E(Y |Z = 1, A = 1)(
E(A|Z = 1)E(P|Z = 1)
)
To recover E(Y|Z = 1, C = 1), we can estimate the left-handside and E(P|Z = 1) directly from the data and use random as-signment to replace E(A|Z = 1) with E(A|Z = 0) and E(Y|Z =1, A = 1) with E(Y|Z = 0, A = 1). That is, we assume treatment-and control-group always-takers are equivalent on average. In ourcase, block randomization means these equalities should also beconditional on block. In practice, calculating them conditionallymakes a trivial difference.
-
20 QUARTERLY JOURNAL OF ECONOMICS
It is possible that there may be some spillover effects of theBAM treatment to other youth within the school, through peer in-fluences or other mechanisms. We have tried to learn more aboutthis using nonexperimental variation across schools in the shareof male youth who were randomized to treatment. But with rela-tively few schools in the study sample our statistical power is quitelimited. If peer spillovers from BAM lead to improved control-group outcomes (or if negative spillovers from interacting with“untreated” control youth undermine effects of the BAM treat-ment), then our estimates would understate the effects of offeringBAM at larger scale (e.g., to all youth within a school rather thanjust some youth).
4. Results. Table IV shows that school engagement in-creased in both studies by the end of the program period. Instudy 1, where we have post-program data, the effect seems tohave persisted. We initially focus on showing as much of the dataas possible, focusing on separate estimates by study and programyear and simple pair-wise p-values without any adjustment formultiple comparisons, but return to these other issues below. Dur-ing the program year in study 1, participation in BAM improvedschooling outcomes by 0.14 std. dev., and by 0.19 std. dev. in thefollow-up year. In study 2 there was no statistically detectableimpact on school engagement in year one but an effect of about0.10 std. dev. in year two (which as a reminder, was a programyear in study 2). Permutation tests lead to qualitatively similarinferences for BAM study 1, although for study 2 the p-value forthe year two schooling effect is now p = .11 vs. p < .05 (see OnlineAppendix Table A.6).
Table IV shows impacts on arrests that look quite similaracross the two studies when measured at the end of the programperiod. By the end of the first (and only) program year in study1, participation reduced total arrests by 28% of the CCM, violent-crime arrests by 45%, and “other” arrests by 37% (with reducedweapons offenses, trespassing, and vandalism each accountingfor about one-quarter of the effect).19 By the end of the programperiod in study 2 (i.e., during year two), participation reducedtotal arrests by 35% of the CCM, violent-crime arrests by 50%,
19. Disorderly conduct and disobeying a police officer—the offenses wherewe might expect being able to interact constructively with police could have thebiggest effect on the chance of being arrested conditional on engaging in a givenbehavior—do not change in either study.
file:qje.oxfordjournals.orgfile:qje.oxfordjournals.org
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 21
TA
BL
EIV
BE
CO
MIN
GA
MA
NS
TU
DIE
S1
AN
D2
–E
FF
EC
TS
ON
YO
UT
HO
UT
CO
ME
S
Con
trol
mea
nIn
ten
tion
totr
eat
Eff
ect
ofpa
rtic
ipat
ion
(IV
)C
ontr
olco
mpl
ier
mea
n
BA
Mst
udy
1(P
rogr
amYe
ar20
09–1
0,n
=2,
740)
Year
1(p
rogr
amof
fere
d)S
choo
len
gage
men
tin
dex
00.
0569
∗∗∗
0.13
67∗∗
∗0.
222
(0.0
215)
(0.0
511)
Tot
alar
rest
spe
ryo
uth
per
year
0.69
9−0
.077
8∗−0
.186
9∗0.
672
(0.0
456)
(0.1
087)
Vio
len
t0.
167
−0.0
345∗
∗−0
.082
9∗∗
0.18
6(0
.016
5)(0
.039
4)P
rope
rty
0.07
70.
0048
0.01
160.
066
(0.0
127)
(0.0
303)
Dru
g0.
151
0.00
130.
0032
0.09
7(0
.017
7)(0
.042
2)O
ther
0.30
5−0
.049
5∗−0
.118
8∗0.
323
(0.0
272)
(0.0
648)
Year
2(p
rogr
amn
otof
fere
d)S
choo
len
gage
men
tin
dex
00.
0782
∗∗∗
0.18
78∗∗
∗0.
040
(0.0
215)
(0.0
514)
Tot
alar
rest
spe
ryo
uth
per
year
0.59
5−0
.064
3−0
.154
30.
606
(0.0
420)
(0.1
000)
Vio
len
t0.
110
0.00
060.
0013
0.09
2(0
.014
3)(0
.034
0)P
rope
rty
0.05
7−0
.003
4−0
.008
20.
052
(0.0
103)
(0.0
245)
Dru
g0.
164
−0.0
196
−0.0
471
0.17
3(0
.019
4)(0
.046
1)O
ther
0.26
4−0
.041
8−0
.100
40.
288
(0.0
259)
(0.0
617)
-
22 QUARTERLY JOURNAL OF ECONOMICST
AB
LE
IV( C
ON
TIN
UE
D)
Con
trol
mea
nIn
ten
tion
totr
eat
Eff
ect
ofpa
rtic
ipat
ion
(IV
)C
ontr
olco
mpl
ier
mea
n
BA
MS
tudy
2(P
rogr
amYe
ars
2013
–14
and
2014
–15,
n=
2,06
4)Ye
ar1
(pro
gram
offe
red)
Sch
oole
nga
gem
ent
inde
x0
0.00
580.
0117
0.22
1(0
.024
8)(0
.048
8)T
otal
arre
sts
per
you
thpe
rye
ar0.
591
−0.0
806
−0.1
614
0.63
0(0
.050
6)(0
.099
9)V
iole
nt
0.11
9−0
.018
0−0
.036
10.
121
(0.0
161)
(0.0
318)
Pro
pert
y0.
073
−0.0
078
−0.0
157
0.07
5(0
.012
9)(0
.025
3)D
rug
0.12
6−0
.015
3−0
.030
70.
168
(0.0
233)
(0.0
459)
Oth
er0.
273
−0.0
394
−0.0
789
0.26
6(0
.029
3)(0
.057
9)Ye
ar2
(pro
gram
offe
red)
Sch
oole
nga
gem
ent
inde
x0
0.05
01∗∗
0.09
93∗∗
0.08
1(0
.025
2)(0
.049
0)T
otal
arre
sts
per
you
thpe
rye
ar0.
383
−0.0
841∗
∗−0
.167
0∗∗
0.47
1(0
.039
2)(0
.077
1)V
iole
nt
0.07
9−0
.027
6∗−0
.054
9∗0.
110
(0.0
155)
(0.0
303)
Pro
pert
y0.
046
−0.0
018
−0.0
036
0.06
2(0
.010
1)(0
.019
7)D
rug
0.09
4−0
.014
7−0
.029
20.
115
(0.0
171)
(0.0
335)
Oth
er0.
163
−0.0
400∗
−0.0
793∗
0.18
3(0
.022
1)(0
.043
4)
Not
es.B
asel
ine
cova
riat
esan
dra
ndo
miz
atio
nbl
ock
fixe
def
fect
sin
clu
ded
inal
lm
odel
spec
ifica
tion
s(s
eete
xt).
Het
eros
ceda
stic
ity-
robu
stst
anda
rder
rors
inpa
ren
thes
es.S
choo
len
gage
men
tin
dex
iseq
ual
toan
un
wei
ghte
dav
erag
eof
days
pres
ent,
GP
A,
and
enro
llm
ent
stat
us
aten
dof
sch
ool
year
,al
ln
orm
aliz
edto
Z-s
core
form
usi
ng
con
trol
grou
p’s
dist
ribu
tion
.Ye
ar1
arre
stda
tafr
omst
art
ofpr
ogra
msc
hoo
lye
aru
nti
lst
art
offo
llow
ing
sch
ool
year
for
both
stu
dies
.F
orst
udy
1,th
eye
ar2
arre
stda
taru
ns
thro
ugh
July
18(c
aptu
rin
ga
∼10
mon
thw
indo
w)
wh
ile
for
stu
dy2,
year
2ar
rest
data
run
thro
ugh
Mar
ch31
(∼8
mon
ths)
.∗p
<.1
0,∗∗
p<
.05,
∗∗∗
p<
.01.
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 23
and other arrests by 43% (driven by reductions in reckless con-duct and trespassing). These effects translate into large numbersof arrests averted, given that the control groups’ rates of arrestswere 40 to 70 arrests per 100 youth per year. (The rate was some-what lower in year two of study 2 because those arrest data coveronly eight months). Some of these impacts are not very preciselyestimated and are sometimes not statistically significant evenwhen the effects are proportionally large. But the consistency inthe pattern of arrest impacts across the studies is striking.
Table V reports the results of pooling together the data fromthe program periods in these two RCTs (year one for study 1 andyears one and two for study 2). The final column shows that wecannot reject the null hypothesis that the effects were the sameacross the two studies.20 In this case pooling data from the twostudies can also improve statistical power. The p-values for pair-wise treatment-control comparisons are statistically significant atthe 5% cutoff for school engagement, total arrests, and arrests forviolent and other crimes, calculated from either robust standarderrors or a permutation test. These results basically remain sta-tistically significant (p-values range from .010 to .055) when wecontrol for multiple comparisons—either the fairly conservativefamily-wise error rate, or FWER (defined as the chance that atleast one of our outcomes in the “family” of outcomes is signifi-cant when the null hypothesis of no effect is true),21 or the falsediscovery rate, or FDR (the share of significant estimates that areexpected to be false positives).22 In Table V we do not includetotal arrests in the “family” of outcomes since that is just a
20. Since the arrest data for year two of BAM study 2 do not cover a fullyear, we upweight these to 12-month equivalents for calculating the final column’sp-values for the test of comparability of study 1 and 2 effects. The point estimatesshown in the table come from summing the arrests over the 19-month periodcovered by study 2 years one and two and then dividing by two.
21. We use a bootstrap resampling technique that simulates data under thenull hypothesis (Westfall and Young 1993). Within each permutation, we randomlyreassign the treatment indicator with replacement and estimate program impactson all five of our main outcomes (the schooling index and our four arrest categories).By repeating this procedure 100,000 times, we create an empirical distributionof t-statistics that allows us to compare the actual set of t-statistics we find towhat we would have found by chance under the null; see Online Appendix C andTable A.5 for details.
22. We find the smallest FDR q-value at which we could reject the nullfor each outcome using the method from Benjamini and Hochberg (1995);Online Appendix Table A.5 shows similar results if we use the method fromBenjamini, Krieger, and Yekutieli (2006).
file:qje.oxfordjournals.orgfile:qje.oxfordjournals.org
-
24 QUARTERLY JOURNAL OF ECONOMICST
AB
LE
VB
EC
OM
ING
AM
AN
PO
OL
ED
ST
UD
IES
1A
ND
2:E
FF
EC
TS
ON
YO
UT
HO
UT
CO
ME
S
Poo
led
prog
ram
effe
cts
H0:P
rogr
amef
fect
=0
H0:S
tudy
1ef
fect
=S
tudy
2ef
fect
Con
trol
mea
nIn
ten
tion
totr
eat
Eff
ect
ofpa
rtic
ipat
ion
(IV
)
Con
trol
com
plie
rm
ean
Un
adju
sted
p-va
lue
Fals
edi
scov
ery
rate
con
trol
(q-v
alu
e)
Fam
ily-
wis
eer
ror
rate
con
trol
(p-v
alu
e)U
nad
just
edp-
valu
e
Sch
ool
00.
0398
∗∗0.
0880
∗∗∗
0.20
3.0
100.
028
.055
.358
enga
gem
ent
(0.0
155)
(0.0
338)
Tot
alar
rest
spe
ryo
uth
per
year
0.60
3−0
.072
7∗∗
−0.1
611∗
∗0.
601
.019
-.6
59(0
.031
0)(0
.068
3)V
iole
nt
0.13
6−0
.026
9∗∗
−0.0
597∗
∗0.
148
.013
0.02
8.0
55.8
23(0
.010
9)(0
.023
9)P
rope
rty
0.06
90.
0026
0.00
580.
064
.751
0.75
1.9
09.4
25(0
.008
2)(0
.018
1)D
rug
0.13
2−0
.004
8−0
.010
60.
116
.701
0.75
1.9
09.5
09(0
.012
4)(0
.027
3)O
ther
0.26
6−0
.043
6∗∗
−0.0
966∗
∗0.
273
.016
0.02
8.0
55.9
37(0
.018
2)(0
.040
0)
Not
es.n
=48
04(a
llob
serv
atio
ns
from
stu
dies
1an
d2
pool
edto
geth
er).
Bas
elin
eco
vari
ates
and
ran
dom
izat
ion
bloc
kfi
xed
effe
cts
incl
ude
din
allm
odel
s(s
eete
xt).
Sta
nda
rder
rors
(in
pare
nth
eses
)cl
ust
ered
onin
divi
dual
sto
acco
un
tfo
rth
etw
ost
ude
nts
wh
oar
ein
both
stu
dies
.T
he
pool
edva
riab
les
capt
ure
the
prog
ram
year
s:ye
ar1
for
stu
dy1
and
year
s1
and
2co
mbi
ned
for
stu
dy2.
For
stu
dy2,
the
com
bin
edsc
hoo
lin
gin
dex
isan
aver
age
ofth
ein
dex
acro
ssth
etw
opr
ogra
mye
ars,
and
the
com
bin
edar
rest
sar
ean
aver
age
acro
ssth
eav
aila
ble
data
inye
ars
1an
d2
(su
mov
er19
mon
ths
ofar
rest
s/2)
.To
acco
un
tfo
rth
edi
ffer
ent
nu
mbe
rof
mon
ths
cove
red
byth
ear
rest
data
,we
test
equ
alit
yac
ross
the
two
stu
dies
byex
trap
olat
ing
the
mon
thly
rate
ofof
fen
din
gto
a12
-mon
thpe
riod
.∗p
<.1
0,∗∗
p<
.05,
∗∗∗
p<
.01.
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 25
TABLE VIBECOMING A MAN STUDY 1: EFFECTS ON HIGH SCHOOL GRADUATION
High school graduationmeasures
Controlmean
Intentionto treat
Effect ofparticipation
(IV)
Controlcomplier
mean
Graduated on time 0.339 0.0297∗ 0.0714∗ 0.382(0.0161) (0.0383)
Ever graduated 0.414 0.0240 0.0577 0.467(transfers = dropouts) (0.0167) (0.0397)
Ever graduated 0.582 0.0355∗∗ 0.0853∗∗ 0.587(transfers = graduates) (0.0170) (0.0406)
Notes. n = 2,740. Table measures graduation from Chicago Public Schools (CPS). First row counts gradua-tion as receipt of diploma on time relative to grade at time of randomization, second and third rows measuregraduation status by end of our study period (spring 2015), first assuming anyone who left the district didnot graduate (second row) then assuming all 474 verified out-of-district transfers did graduate (third row).Baseline covariates and randomization block fixed effects included in all models (see text). Heteroscedasticity-robust standard errors in parentheses. ∗ p < .10, ∗∗ p < .05, ∗∗∗ p < .01.
linear combination of the crime-specific measures (Online Ap-pendix Table A.5 shows the results from using different ways ofdefining “families” of outcomes).
The online appendix shows that the results are robust to ad-justing for the possibility of under-reporting of sports participationin the first BAM study (Online Appendix Table A.4), using multi-ple imputation or other methods to deal with missing values forschool outcomes (Online Appendix Table A.10), and using differ-ent thresholds for deciding what counts as a “match” to the arrestdata in our probabilistic matching algorithms (Online AppendixTable A.7).
Finally Table VI shows that there seem to have been gainsin high school graduation in the first BAM study (we do not yethave follow-up data for study 2). While the size and p-values ofthe estimated impacts vary somewhat depending on the gradua-tion measure, all estimates are in the direction of sizable gainsin graduation rates. The narrowest definition of graduation is ob-taining a diploma on time (no delays relative to the grade levelduring the program year), which BAM participation increased by7 percentage points (p < .10). Given a CCM of 38%, this is anincrease of 19%. The broadest definition is ever having received adiploma from CPS. This measure requires an assumption abouthow to treat youth who leave the school district, since for themwe do not observe graduation or dropout. The program effect wasslightly smaller (6 percentage points, versus a CCM of 47%, for
file:qje.oxfordjournals.orgfile:qje.oxfordjournals.orgfile:qje.oxfordjournals.orgfile:qje.oxfordjournals.orgfile:qje.oxfordjournals.org
-
26 QUARTERLY JOURNAL OF ECONOMICS
a 12% increase) and not quite statistically significant if we counttransfers out of CPS as dropouts, but larger and statistically sig-nificant if transfers are counted as graduates (nearly 9 percentagepoints, p < .05, versus CCM of 59%, for a 15% increase). Increasedgraduation rates should lead to gains in lifetime earnings and im-proved health. We discuss this in the conclusion in the context ofour rough benefit-cost analysis.
III.B. Study 3: Cook County Juvenile TemporaryDetention Center
1. Sample and Randomization. Our third RCT capitalizeson a natural experiment resulting from the May 2007 take-overof the JTDC by a federal judge as the result of an ACLU lawsuit(Doe v. Cook County 2007). One of the first acts of Earl Dunlap, thetemporary administrator who was appointed by the federal courtto run the 500-bed facility, was to divide the facility into 10 essen-tially separate residential centers of around 50 beds each and toenact the reforms described in Section II in each of these centersone by one. The rollout of the reforms across centers was haltedhalfway through due to litigation initiated by the union represent-ing the JTDC staff. The result was that for an extended period,half the JTDC centers operated using the reforms (“treatmentcenters”) while the rest used the previous standard operating pro-cedures (“control centers”).
Our research team worked with the JTDC staff to implementa randomization algorithm that assigned all incoming male youthto treatment or control centers from November 10, 2009, throughMarch 2, 2011. The randomization ended when the litigation wasresolved, at which point the entire facility began implementing thenew reforms. (Girls were not randomized because all girls werehoused in a single residential center.) In our main analyses wefocus on the 2,693 male admissions to the JTDC during our studyperiod for which we have at least 18 months of follow-up data, sothat we have a balanced panel for the full follow-up period. Resultsfor all 5,728 male admissions to the JTDC during our study periodare in the Online Appendix (see Tables A.17 through A.20).
2. Estimation Approach. While random assignment was notbinding for some youth because of safety or operational reasons,
file:qje.oxfordjournals.org
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 27
or because they had been assigned to a treatment unit inside theJTDC previously (see Online Appendix B), randomization greatlyincreased the likelihood of placement in a treatment unit. The ITTeffect of random assignment on placement is about 25 percentagepoints (39.5% in spells where youth are assigned to treatment,14.4% for controls); the first stage F-statistic is 241. We thus havean “encouragement design,” where randomization is a valid in-strument for estimating the effect of participation on compliersas in equations (2) and (3) above. To avoid mis-measurement oftreatment caused by temporary relocation of small-group livingfacilities,23 we define participation as spending at least 5% of aJTDC stay in a treatment center. The first-stage ITT changes little(from 25 to 24 points) if participation is instead counted as “ever”in a treatment center in a given spell. In addition to controllingfor baseline characteristics, because randomization occurred foryouth by day of admission we also control for day-of-admit fixedeffects. These help control for any slight differences across daysin treatment assignment probabilities, and may also help withprecision (see Online Appendix C).
3. Data and Outcome Measures. The data we have on theseyouth include intake forms that provide basic demographics andaddresses; admissions logs, which the admissions staff use torecord who enters the facility each day; and the JTDC’s hous-ing roster, which captures the residential unit in which a youth islocated on each day and so lets us measure receipt of treatment.We have these data through December 2011. Our main resultsfocus on a common measure of recidivism—readmission into theJTDC facility itself. We have also linked these youth to the CPDand ISP arrest databases using the same probabilistic matchingalgorithms described above.
Table VII makes clear that this is a very criminally in-volved population: the average youth has been arrested eight
23. In general, we observe whether a youth participated in CBT by observingwhether he lived in an area of the JTDC that offered CBT at the time of hisstay. Occasionally, the youth and staff in a small-group living facility, or “pod,”temporarily moved to other areas of the building due to maintenance, cleaning, orother facility issues. We have imperfect records on these temporary moves, whichintroduces some error in our measurement of CBT receipt.
file:qje.oxfordjournals.orgfile:qje.oxfordjournals.org
-
28 QUARTERLY JOURNAL OF ECONOMICS
TABLE VIIJUVENILE TEMPORARY DETENTION CENTER STUDY 3: BASELINE CHARACTERISTICS
Baseline characteristic Control Treatment
No. of spells = 1,322 No. of spells = 1,371Demographics
Age 16.13 16.15Black 0.83 0.84Hispanic 0.14 0.12White 0.03 0.03Other 0.01 0.01
No. of prior JTDC spells includingfocal spell
3.35 3.19
Reason for JTDC admissionViolent crime 0.17 0.18Property crime 0.10 0.09Drug crime 0.08 0.07Other crime 0.35 0.35Directly admitted (no police arrestrecorded)
0.30 0.30
Number of prior arrestsAll offenses 8.25 7.85∗
Violent offenses 2.11 2.00Property offenses 1.55 1.49Drug offenses 1.34 1.25Other offenses 3.22 3.08
Neighborhood characteristicsPercent adults 25+ with at least HSdiploma
72.80 72.75
Percent black 69.13 68.12Percent Hispanic 17.71 18.44Percent below poverty 34.54 34.58Unemployment rate 18.59 18.58
p-value on F-test of treatment-controlcomparison for all baselinecharacteristics
.65
Notes. Asterisks indicate statistical significance of pairwise treatment-control comparison for a given base-line characteristic, clustering standard errors on individual and clustering on census tract for the neighbor-hood characteristics. Neighborhood characteristics come from geocoding address information and linking totract level data from the American Community Survey. Joint significance test for equality of all baselinecharacteristics uses only nonmissing data (n = 2,643) and clusters standard errors on individual. ∗ p < .10,∗∗ p < .05, ∗∗∗ p < .01.
times in the past. The average JTDC spell in our sampleis the youth’s third entry into detention.24 Consistent with
24. Among the 1,862 individuals who make up these 2,693 spells, each visitsthe JTDC an average of 4.4 times before the end of our data (the maximum totalspells per individual over the 7 years of our housing roster data is 23).
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 29
national patterns of incarceration, the large majority of de-tainees is black despite the fact that just one-quarter of thecounty’s population is black.25 As we would expect with success-ful randomization, we cannot reject the null hypothesis that thetreatment-control differences are jointly zero (F(24, 1835) = 0.87,p = .65).26
4. Results. Figure I shows the effect of being in a treatmentcenter on the probability of readmission to the JTDC.27 We mea-sure readmission at different points in time since release fromthe JTDC. The first panel shows that two months after release,the ITT effect was a decline in readmission rates of about threepercentage points. Through 18 months the effect was about 4 per-centage points. The figure also shows our estimates are not verysensitive to inclusion of day-of-JTDC-admission fixed effects.
The bottom panel of Figure I shows that the effect of beingin a treatment unit on the compliers (the local average treatmenteffect) was about 13 percentage points two months after releaseand grew slightly to 16 percentage points by 18 months followingexit from the JTDC (equal to 39% and 21% of the CCMs, respec-tively). A different way to gauge the size of this effect is to notethat about a fifth of the control compliers had not been readmit-ted to the JTDC within 18 months; the treatment increased thechances of avoiding readmission by fully 80%.
Table VIII shows results for the effect of receiving the inter-vention (LATE) on the re-admission outcome reported in Figure I
25. http://quickfacts.census.gov/qfd/states/17/17031.html26. Baseline covariates in the joint test are: spell number; age; number of
each type of baseline arrest (violent, property, drug, other); indicators for race(white, Hispanic, other); type of admitting offense (violent, property, drug, or otherarrest, or direct admission with no arrest); and neighborhood characteristics fromthe ACS (unemployment, median income, % below poverty, % white, % black, %Hispanic, % receiving SNAP, % owning their own home, % on welfare, and % withat least a high school degree). Only nonmissing covariates are used in the jointtest. For outcome regressions, we impute zeros for missing values and includeindicator variables for missing-ness. Outcome baseline covariates are: dummiesfor ages 14–15, 15–16, and over 17; race/ethnicity; having one, two, or three-plusprior arrests of each type; neighborhood characteristics (% with at least a highschool degree, % black, % unemployed); indicators for spell number (2, 3, 4, and5 or over); and admission reason (violent, property, drug, or other arrest, with noarrest as the left-out category).
27. Aizer and Doyle (2015) find large negative effects of a similar form ofjuvenile incarceration on high school completion and large positive effects on sub-sequent adult incarceration.
http://quickfacts.census.gov/qfd/states/17/17031.html
-
30 QUARTERLY JOURNAL OF ECONOMICS
FIGURE I
Effect of Treatment on Readmission, Study 3 (Juvenile Detention)
Sample consists of the N = 2,693 youth admitted to the Cook County JTDCduring period when random assignment was in effect, and for whom we haveat least 18 months of follow-up data. Since randomization occurred by day ofadmission, day-of-admit fixed effects help control for any slight differences acrossdays in treatment assignment probabilities. Graph shows effects conditional onbaseline covariates as described in text, and with versus without day-of-admissionfixed effects. Error bars represent 95% confidence intervals. CM is control mean;CCM indicates control complier mean.
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 31
TA
BL
EV
III
JTD
CS
TU
DY
3–
TR
EA
TM
EN
TE
FF
EC
TO
NM
EA
SU
RE
SO
FR
EC
IDIV
ISM
WIT
HIN
GIV
EN
NU
MB
ER
OF
MO
NT
HS
SIN
CE
RE
LE
AS
E;S
AM
PL
EO
FY
OU
TH
WIT
HC
OM
PL
ET
E18
-MO
NT
HF
OL
LO
W-U
PD
AT
A
Mon
ths
sin
cere
leas
e2
46
810
1214
1618
Pan
elA
:Eff
ect
onpr
obab
ilit
yof
read
mis
sion
byX
mon
ths
Eff
ect
ofpa
rtic
ipat
ion
(IV
)−0
.134
2∗∗ −
0.18
93∗∗
∗ −0.
1984
∗∗∗ −
0.20
10∗∗
∗ −0.
1764
∗∗−0
.168
9∗∗ −
0.19
11∗∗
∗ −0.
1598
∗∗−0
.164
3∗∗
(0.0
682)
(0.0
734)
(0.0
735)
(0.0
722)
(0.0
704)
(0.0
690)
(0.0
683)
(0.0
673)
(0.0
669)
Con
trol
com
plie
rm
ean
0.34
50.
563
0.68
10.
738
0.73
30.
768
0.79
30.
790
0.79
5
Pan
elB
:Eff
ect
onn
um
ber
ofti
mes
you
thre
adm
itte
dw
ith
inX
mon
ths
Eff
ect
ofpa
rtic
ipat
ion
(IV
)−0
.163
3∗−0
.329
5∗∗∗
−0.3
411∗
∗−0
.473
9∗∗∗
−0.5
353∗
∗∗−0
.528
6∗∗ −
0.69
11∗∗
∗ −0.
7031
∗∗∗ −
0.70
45∗∗
∗(0
.083
5)(0
.123
5)(0
.149
3)(0
.174
6)(0
.195
6)(0
.214
8)(0
.233
1)(0
.246
4)(0
.252
2)C
ontr
olco
mpl
ier
mea
n0.
387
0.81
61.
041
1.38
61.
578
1.73
41.
995
2.11
72.
196
Pan
elC
:Eff
ect
onn
um
ber
ofti
mes
you
thre
adm
itte
dor
rear
rest
edw
ith
inX
mon
ths
Eff
ect
ofpa
rtic
ipat
ion
(IV
)−0
.181
9−0
.332
3∗−0
.265
6−0
.403
6−0
.543
4∗−0
.568
1−0
.675
7∗−0
.661
0−0
.659
5(0
.116
8)(0
.180
3)(0
.224
7)(0
.270
2)(0
.319
1)(0
.357
4)(0
.401
9)(0
.438
2)(0
.461
5)C
ontr
olco
mpl
ier
mea
n0.
653
1.34
61.
855
2.48
63.
063
3.49
73.
998
4.42
04.
803
Not
es.
n=
2,69
3.T
able
pres
ents
resu
lts
from
lin
ear
prob
abil
ity
mod
elw
ith
depe
nde
nt
vari
able
indi
cati
ng
wh
eth
eryo
uth
retu
rned
toJT
DC
wit
hin
Xm
onth
s,h
owm
any
tim
esth
eyre
turn
ed,
and
wh
eth
erth
eyre
turn
edor
wer
ere
arre
sted
wit
hin
Xm
onth
s;av
erag
em
argi
nal
effe
cts
from
prob
itm
odel
are
sim
ilar
.S
ome
indi
vidu
als
hav
em
ult
iple
spel
lsan
dpr
ovid
em
ult
iple
data
poin
tsin
the
sam
ple.
We
pres
ent
robu
stst
anda
rder
rors
clu
ster
edon
the
indi
vidu
al.
LA
TE
op-
erat
ion
aliz
espa
rtic
ipat
ion
assp
endi
ng
mor
eth
an5%
ofa
stay
ina
CB
Tu
nit
.A
llre
gres
sion
sin
clu
deba
seli
ne
cova
riat
esan
dda
y-of
-adm
issi
onfi
xed
effe
cts.
∗p
<.1
0,∗∗
p<
.05,
∗∗∗
p<
.01.
-
32 QUARTERLY JOURNAL OF ECONOMICS
and several other outcomes as well (ITT results are inOnline Appendix Tables A.13 through A.20). The second panelof the table shows that receipt of the JTDC intervention reducedthe number of readmissions through 18 months by 0.7 admissionsper youth, equal to 32% of the CCM. The bottom panel showsthat when we combine re-admissions and arrests as our outcome,the point estimate is of about the same magnitude (0.66), butless precisely estimated. The appendix presents various sensitiv-ity analyses, such as what happens when we use the full sample,not just those for whom we have a full 18 months of follow-up data(Online Appendix Tables A.17 through A.20).
While the outcome measures we have available are not per-fectly consistent across all three studies, to the extent to whichwe can pool data the impacts on the one outcome we can examinein all three RCTs (any criminal behavior) seem to be generallyconsistent. We find a statistically significant effect when averag-ing all three studies together and cannot reject the null hypoth-esis that the effects are the same across the three studies (seeOnline Appendix Table A.21).
IV. MECHANISMS
Why do these interventions have such large behavioral im-pacts? In this section we first discuss which specific componentsof the programs may be most important in generating behavioralresponses, and then turn to a discussion of what specific channelsor mechanisms the program components may be working throughto change youth outcomes. Existing theories of the determinants(beyond academic skills) of people’s life outcomes suggest a num-ber of candidate channels. Partly because these candidate mech-anisms map only imperfectly to the specific activities included inthe BAM and JTDC interventions, we also develop an alternativehypothesis (automaticity) for why these programs may changeyouth outcomes. While the data we have to test these mecha-nisms are not totally ideal, we find no positive evidence in supportof some commonly discussed determinants of youth behavior andsocial-program success. We do find some evidence to support ourautomaticity theory.
IV.A. Program Components
Because both the BAM and JTDC programs we study arebundled interventions, we begin by considering which elements of
file:qje.oxfordjournals.orgfile:qje.oxfordjournals.orgfile:qje.oxfordjournals.org
-
FIELD EXPERIMENTS TO REDUCE CRIME AND DROPOUT 33
these interventions may be most important in changing youth out-comes. One natural question is whether these results are simplydriven by the after-school sports programming. Since some pro-grams try to reduce delinquency by keeping youth busy and offthe streets, to what degree are the impacts we observe here dueonly to “voluntary incapacitation” of youth during the after-schoolhours? We can rule out this sort of explanation by using exactdate of arrest in the rap sheet data we have; the estimated effectof BAM on arrests was not concentrated on days when after-schoolprogramming was held.28
In principle, the sports programming could change youth out-comes through other means, such as physical exercise, increasedself-discipline, or delivery of BAM principles in a different format(coaches were trained in BAM principles). But this seems veryunlikely to be the main driver behind our estimated program im-pacts. In the second BAM study, sports participation declined by80% from the first to second year of the program,29 while the im-pact on violent-crime arrests increased from year one to year twoby over 50%. Perhaps more telling, the JTDC study generated siz-able changes in behavior despite including no sports componentat all.
The JTDC intervention was a different bundle of services thatinvolved other program elements beyond CBT such as use of atoken economy inside the facility’s treatment units and exposureto staff with somewhat higher educational qualifications. Whilewe cannot cleanly distinguish between these elements, the factthat our behavioral impacts in the JTDC study are all measuredduring the time period after youth left the facility does not seemconsistent with the idea that changes in incentives inside thefacility (like the token economy) are driving our results.
IV.B. Candidate Mechanisms
1. Candidates from Existing Research. Table IX describesthe different candidate mechanisms through which the program-
28. In BAM study 1 the ITT effect on an indicator for any violent-crime arreston days when after-school programming was not offered was β = −0.0224 (std. err.= 0.0103), p = 0.030, CM = 0.094, versus on days when after-school programmingwas offered, β = −0.0061 (0.0076), p = 0.423, CM = 0.046). These estimates donot adjust for the larger number of non-programmi
top related