boys & girls clubs and school collaborations: a longitudinal study of a multicomponent substance...

BOYS & GIRLS CLUBS ANDSCHOOL COLLABORATIONS:A LONGITUDINAL STUDY OF AMULTICOMPONENT SUBSTANCEABUSE PREVENTION PROGRAMFOR HIGH-RISK ELEMENTARYSCHOOL CHILDREN

Tena L. St. PierreDepartment of Agricultural and Extension Education, and Institutefor Policy Research and Evaluation,The Pennsylvania State University

Melvin M. MarkDepartment of Psychology, The Pennsylvania State University

D. Lynne KaltreiderInstitute for Policy Research and Evaluation,The Pennsylvania State University

Bernadette CampbellDepartment of Psychology, The Pennsylvania State University

n

This study tested the effectiveness of a multicomponent after-schoolsubstance abuse prevention program for high-risk second- and third-gradechildren implemented collaboratively by Boys & Girls Clubs and theirlocal schools. The 2-year program was designed to reduce risk factorspredictive of later substance abuse, and to enhance protective factorsshown to buffer risk. Results showed positive effects on program children’spersonal competency skills including (1) refusing wrongdoing; (2) solvingpeer and school problems; (3) showing courteousness to teachers and otherschool personnel; and (4) behaving ethically. For the most part, theprogram also had positive effects on children’s feelings toward school and

Correspondence to: Tena L. St. Pierre, Institute for Policy Research and Education, The Pennsylvania StateUniversity, N 253 Burrowes Building, University Park, PA 16802–6201. E-mail: [email protected]

A R T I C L E

JOURNAL OF COMMUNITY PSYCHOLOGY , Vol. 29, No. 2, 87–106 (2001)© 2001 John Wiley & Sons, Inc.

grades in spelling. Results suggest that youth-serving organizations andschools can collaboratively implement multicomponent interventions thatprovide protective factors that may buffer high-risk elementary schoolchildren from the multiple risks in their lives for future drug abuse.© 2001 John Wiley & Sons, Inc.

INTRODUCTION

Great strides have been made in the last decade in the development of effectiveinterventions to prevent youth problem behaviors such as substance abuse ~Howell,Krisberg, Hawkins, & Wilson, 1995; Institute of Medicine @IOM#, 1994!. Particularlypromising are multicomponent intervention programs that seek to reduce risk factorsand enhance protective factors within the multiple, interdependent environments inwhich youth grow and develop ~Coie et al., 1993!. Prevention programs implementedcollaboratively by diverse community organizations can target multiple risk factorsacross youth environments and developmental periods ~Harachi, Ayers, Hawkins, Cat-alano, & Cushing, 1995!. However, despite the view by many prevention scientists thatprevention efforts are more likely to be effective when implemented early in devel-opment, empirical studies of multicomponent programs for elementary school-agechildren are rare ~Coie et al., 1993!.

A well-developed literature exists on risk factors in the individual, family, school,peer relations, and community environments that predict increased likelihood ofsubstance abuse ~e.g., Hawkins, Catalano, & Miller, 1992; Hawkins, Kosterman, Maguin,Catalano, & Arthur, 1997!. There is also growing interest in the role of protectivefactors that may interact with risk factors to buffer exposure to risk ~Hawkins, Arthur,& Catalano, 1995; Rolf, Masten, Cichetti, Neuchterlein, & Weintraub, 1990; Rutter,1985; Werner & Smith, 1992! and in the more recent work by the Search Institute onbuilding developmental assets that children and adolescents need to grow up healthyand competent ~Benson, 1997; Benson, Scales, & Roehlkepartain, 1999!.

In this article, we describe the evaluation of a multicomponent intervention devel-oped and implemented collaboratively by three Boys & Girls Clubs and their localelementary schools. The goal was to prevent future substance abuse and other prob-lem behaviors among high-risk second- and third-grade children by reducing identi-fied risk factors and enhancing protective factors in individual youth, and in theirfamily and school environments. During program development, a number of riskfactors were identified that are relatively common in high-risk environments.

In our study communities, as in many high-risk environments, many childrenpossessed poor social skills, communication, and decision-making skills. They oftenwere described as aggressive, disruptive, and lacking the basic interpersonal skills toget along with others. The literature suggests that behavior problems in the earlyelementary grades are associated with an increased risk of later drug abuse ~Hawkinset al., 1992!. However, social competency promotion interventions for children inelementary school have been shown to improve cognitive and behavioral problem-solving skills, behavior, school adjustment, peer acceptance, and ability to cope withproblems ~Elias & Clabby, 1989, 1992; Schinke & Tepavac, 1995; Weissberg et al.,1981!, providing a protective factor against future substance abuse.

The school environment has a particularly important influence on children, giventhe time spent and the level of academic and social learning that occurs in that

88 • Journal of Community Psychology, March 2001

setting. Although failure in school has been identified as a predictor of adolescentsubstance abuse ~ Jessor, 1976; Robins, 1984!, studies suggest that social adjustment isa more important predictor than academic performance in the early elementarygrades. Early antisocial behavior in school may predict later academic failure ~Feld-husen, Thurston, & Benning, 1973! and substance abuse.

Many of the children in our study apparently were not committed to school, andwere performing poorly. Few parents attended parent0teacher conferences. Somechildren did not attend school routinely, in part because their parent~s! lacked moti-vation to get up in the morning to get the child ready. Children who are not com-mitted to education are more likely to engage in later drug use and delinquentbehavior ~Brooks, Linkoff, & Whiteman, 1977; Elliott & Voss, 1974; Friedman, 1983!,but increasing parent involvement can be a protective factor. Parent involvement hasbeen associated with improved academic effort, grades, and attendance among stu-dents showing low commitment to school ~Bien & Bry, 1980; Blechman, Taylor, &Schrader, 1981!. Individualized tutoring programs also have shown promise for reduc-ing academic failure and problem behaviors ~Hawkins et al., 1992!.

School-based activities that promoted school bonding also were limited in the studycommunities. Many parents had not had successful school experiences, and conse-quently, were not predisposed to view school favorably. Moreover, a lack of communi-cation between the schools and parents appeared to have created a gulf between themand a lack of understanding of one another, making it difficult to nurture a bond ~Comer,1988!. Studies suggest that children may work harder in school and get better grades whenschools involve parents in determining academic and social needs of children, and inproviding assistance to address these needs ~Bien & Bry, 1980; Blechman et al., 1981!.

Family bonding also appeared low in these three communities. Some parentsroutinely told their children to go to the Boys & Girls Club after school and not toreturn home until the Club was closed. In addition, few children appeared to partici-pate in activities with their parents. Poor bonding between children and parents~Brook, Brook, Gordon, Whiteman, & Cohen, 1990!, low parental involvement inactivities with children ~Kandel & Andrews, 1987!, and maternal isolation ~Werner &Smith, 1982! are predictors of later substance abuse. However, strategies to fosterpositive family relationships and to provide opportunities for parents and children toshare positive activities create a protective factor because parent–child involvementand attachment tend to discourage youths’ initiation into substance use ~Brook, Gor-don, & Whiteman, 1986; Jessor & Jessor, 1977!.

Establishing a bond with at least one person, such as a grandparent, teacher, orother adult who is a positive role model, also can buffer youth from risks ~Werner &Smith, 1982!. Outside the family, an external support system that encourages chil-dren’s coping efforts and reinforces positive values may protect against future sub-stance use ~Garmezy, 1985!.

Parents in our target population often did not discipline consistently and seemedindifferent about their children’s progress in school. Poor family management prac-tices ~Coombs & Landsverk, 1988! and low parental educational aspirations for chil-dren ~Kandel & Andrews, 1987! put children at risk. Parenting skills programs to helpparents set clear expectations, consistently reinforce positive behavior, supervise, andsupport children’s social, academic, and refusal skills can serve as a protective factor.These family management skills have been shown to reduce children’s behavior prob-lems in preschool and elementary school, increase academic performance, and helpchildren resist social influences to use substances ~Hawkins et al., 1992!.

Boys & Girls Clubs • 89

Furthermore, the neighborhoods in which the three Boys & Girls Clubs are locatedprovided daily availability of alcohol, tobacco, and other drugs. Children regularlyobserved people using and selling drugs in their neighborhoods ~and sometimes athome!. Because initiation into drug use is preceded by values favorable to its use~Kandel, Kessler, & Margulies, 1978; Krosnick & Judd, 1982; Smith & Fogg, 1978!, itappears helpful for children to learn at an early age the harmful influences of drugsand how to resist the social pressures associated with substance abuse that are allaround them.

BEYOND THE SCHOOLS

Implementing multicomponent interventions that target risk and protective factors inmultiple environments requires coordinated action in each environment being tar-geted. Therefore, it is beneficial to design multiple intervention components thateach address risk factors in different environments. Examples include the often-citedMidwestern Prevention Project ~Pentz et al., 1989!, Communities That Care ~Hawkinset al., 1992!, and the Center for Substance Abuse Prevention’s Community PartnershipGrants. The most common models include a school-based drug prevention curricu-lum for youth with other program elements such as parent skills training, a mediacampaign, community organization task forces, and health policy interventions.

These models appear to assume that the youth skills training components of theinterventions will remain the school’s responsibility because of their “sense of publiclegitimacy” ~Benard, Fafoglia, & Perone, 1987!, their captive audience of young peo-ple, and their traditional role in socializing youth into the larger society ~Perry, 1986!.However, because youth spend the greatest amount of their time outside of school~Task Force on Youth Development and Community Programs, 1992!, and many riskfactors predictive of future substance abuse exist in environments besides the school,some authors believe the school alone should not be held completely responsible forthe problem and the solution ~e.g., Dryfoos, 1998; Kaltreider & St. Pierre, 1995; St.Pierre, Kaltreider, Mark, & Aikin, 1992; St. Pierre, Mark, Kaltreider, & Aikin, 1997!.

Furthermore, many researchers conclude that primary prevention and early inter-vention are most effective when programs target youth at greatest risk, for example,large social groupings such as neighborhoods or schools in which multiple risks exist~e.g., Coie et al., 1993!. Many of these effective programs have been implemented bycommunity agencies and organizations, and have included elements such as earlyintervention, parent involvement, peer involvement, social and life skills training, andattention to staff training and supervision ~Dryfoos, 1998!.

In particular, community-based youth-serving organizations have shown great poten-tial for offering effective prevention programs for high-risk youth ~e.g., St. Pierreet al., 1992, 1997!. More than 17,000 organizations such as Scouts, 4-H, YWCA, YMCA,and Boys & Girls Clubs offer community-based youth programs, many of which includestrategies for youth to acquire life skills and the ability to communicate, make deci-sions, solve problems, and set goals for education and careers ~Task Force on YouthDevelopment and Community Programs, 1992!.

By providing appealing recreational and voluntary nonformal educational pro-grams for youth outside school hours, youth-serving organizations are in an idealposition to partner with schools and parents to provide programs that address theneeds of local youth. One of the recommendations made by the Task Force on YouthDevelopment and Community Programs ~1992, p. 79!, is for community youth pro-


grams to “reach out to families, schools, and a wide range of community partners inyouth development,” especially in low-income neighborhoods. Moreover, younger chil-dren, ages 5 to 9, and children living in low-income neighborhoods have been shownto gain the most from after-school programs, showing better behavior with peers andadults, work habits, and school performance ~The David and Lucile Packard Founda-tion, 1999!.

Youth-serving organizations typically work with youth in small groups, and are ledby adult leaders who serve as positive role models and mentors. Youth participate vol-untarily in enjoyable and challenging activities that are not solely academic, making theexperience very different from school. Such programs may be more appealing for high-risk youth who have not bonded with school. Furthermore, youth-serving organizationsmay be more successful reaching some high-risk parents who themselves have had neg-ative experiences with school. Because of these characteristics, youth-serving organiza-tions can offer programs that complement rather than compete with schools.

In this article, we report results of a multicomponent after-school preventionprogram for second- and third-grade children implemented collaboratively by Boys &Girls Clubs and their local schools. The study was designed to determine, relative to a controlgroup, whether the 2-year program targeting children, schools, and parents would be effective inreducing risk factors predictive of later substance abuse and enhancing protective factors shownto buffer risk.

METHOD

The Multicomponent Intervention

The intervention consisted of three components: a youth component called SMARTKids, a school component called SMART Teachers, and a parent component namedSMART Parents. Each intervention component was designed to reduce specific riskfactors in the individual and in children’s school and family environments discussedabove, and to enhance protective factors that could buffer children from exposure tothese risks. A staff member at each Boys & Girls Club had primary responsibility forcoordinating the program components. Each prevention coordinator was assisted by ahalf-time prevention program assistant.

SMART Kids. The youth component, SMART Kids, was implemented in the three Boys& Girls Clubs 3 hours each day after school while children were in second and thirdgrades. The program offered homework assistance for 30 minutes Monday throughThursday. The program also attempted to provide one-on-one tutoring 1 hour perweek to improve academic performance and promote school bonding. Tutors wererecruited from community residents, local businesses, and nearby college sororitiesand fraternities seeking to do community service. Some teachers from collaboratingschools also volunteered to tutor. The after-school program included a snack andrecreational activities when children were not in the homework program or with atutor. Children had access to all Boys & Girls Clubs activities at other times.

Also offered was a 2-year developmentally appropriate small-group preventionprogram called SMART Kids. The 10-week prevention curriculum, offered by theprevention coordinator once a week for 30–45 minutes, is designed to enhance youn-ger children’s personal and social competence by teaching broad skills for coping withlife. Specifically, the program aims to ~a! encourage self-esteem and self-responsibility;


~b! develop positive social skills; ~c! develop critical thinking and decision-makingskills; and ~d! help protect children from substances and other harmful influences.SMART Kids was adapted by Boys & Girls Clubs of America from the Million DollarMachine ~Davis & GoWell, 1990!, a school-based curriculum found effective in improv-ing self respect, responsibility, and environmental awareness with elementary schoolchildren ~Schinke & Tepavac, 1995!.

The after-school youth component included two enrichment activities each schoolyear such as theater performances and trips to art and science museums. Theseactivities were designed to expose children to new experiences and to promote groupcohesion, interaction with a positive peer group, and continuous involvement in theprogram.

SMART Teachers. The overall goal of the school component was to forge a partnershipbetween the schools and Boys & Girls Clubs to address systematically the academic andsocial needs of program children. A volunteer teacher at each school served as thecontact teacher to help ensure that program-related school activities were accomplished.

The Boys & Girls Club prevention coordinators served as a link between parentsand schools, sometimes persuading parents to attend parent0teacher conferences,attending such conferences with parents when requested, and sometimes providing trans-portation. Prevention coordinators met monthly with program children’s teachers toassess children’s academic and social progress and to determine how to address eachchild’s needs in the after-school program. As appropriate, this information was sharedwith after-school program assistants and individual tutors at the Boys & Girls Clubs.

SMART Parents. The parent component consisted of a five-session program calledSMART Parents developed by the investigators. It was offered at each Boys & GirlsClub in informal discussion format and was cofacilitated by the Boys & Girls Clubprevention coordinator and the contact teacher. The session topics, which were designedto help parents help their children succeed in school, to increase school bonding, andgenerally to provide social support among program parents, were: ~1! Establishing aDaily Routine; ~2! Helping Your Child Succeed with Homework; ~3! Having a Success-ful Parent0Teacher Conference; ~4! Improving Your Child’s Behavior in School; and~5! Reading with Your Child. Cofacilitation of the parent meetings by the preventioncoordinator and the school contact teacher was intended to demonstrate the school0Club linkage as advocates for program children and parents. Parents were givenone-page colorful fact sheets with key points for each session that they were encour-aged to hang on their kitchen refrigerators. At the end of each meeting, parents weregiven a small laminated card titled “Tips for Parents.” Each card contained a shortmessage describing one thing parents can do to help prevent children’s future sub-stance abuse. Examples of “Tips for Parents” are “Be a Good Role Model or Example,”“Spend Time with Your Child,” and “Help Your Child Resist Pressure.”

Parents also were encouraged to assist with the after-school program and to attendparent0teacher conferences. Family social activities were held several times a year toencourage family bonding. Activities included family movies, bowling and skatingoutings, and dinners at the Boys & Girls Clubs.

DESIGN AND PROCEDURES

Three Boys & Girls Clubs from three different states in the East and South partici-pated in the study. Purposive sampling was used to select sites. The three Boys & Girls


Clubs had served as demonstration sites in two previous longitudinal substance abuseprevention studies and were committed to substance abuse prevention as an organi-zational goal. The Clubs had experience implementing prevention programs, includ-ing one that contained a parent involvement component ~St. Pierre et al., 1997!. Theywere familiar with strategies for recruitment and retention of subjects for a researchstudy, had successfully met the requirements of prior research projects, and had along-standing working relationship with the researchers. The five collaborating ele-mentary schools served the same children as their local Boys & Girls Club.

The study employed a randomized experiment with a pretest and multiple post-tests. At each Club location, children were recruited at the end of first grade. Recruitedchildren were rated by their teachers on general math and language skills. Based onthese ratings, children were blocked into overall skill groups for each school and,within blocks, randomly assigned to either the program or the control condition. Theprogram was offered for 2 years during children’s second and third grade, and addi-tional follow-up data were collected in fourth grade. Pretest data were collected inNovember of 1997, immediately prior to the initiation of the program. The secondwave of data collection was 6 months later ~May 1998! at the end of second grade, withsubsequent waves of data collection in the fall and spring of third grade ~November1998 and May 1999!. Follow-up data were collected in the fall of fourth grade ~Novem-ber 1999!. Over 18 months, beginning in November of 1997, program children wereoffered all components of the program described above. Control group children werenot offered any of these program components.

Participants

Prior to recruitment of children at each site, the Boys & Girls Club prevention coor-dinator, Club director, and the investigators held a meeting at the school with teachersand counselors to explain the program and to elicit suggestions for its developmentand implementation. Emphasis was placed on creating a team approach with teachersto address children’s needs, and on minimizing burdens the project might place onteachers.

The prevention coordinator at each of the Boys & Girls Clubs visited first-gradeclassrooms of their collaborating elementary school~s! in April of 1997 to invite allchildren to participate in the study. Children subsequently were taken by their teach-ers on a field trip during the school day to the Boys & Girls Club. During this trip,they met the staff, toured the Club, and took part in age-appropriate recreationalactivities. Upon leaving the Boys & Girls Club, children took home a letter of invita-tion and request for parental consent for the parent and child to take part in thestudy. Parents were informed that they and their child had an equal chance of beingassigned to the program or control group, but that their child would receive a freeBoys & Girls Club membership regardless of group assignment.

Boys & Girls Club prevention coordinators made as many as three waves of homevisits to all potential recruits. They first made one or more home visits to the familiesnot returning consent forms. They made a second wave of home visits over thesummer to remind recruited families that their participation would begin in the fall.When fall arrived, prevention coordinators visited families once more to tell them thegroup to which they had been randomly assigned. They also reminded them thatchildren would receive a free Boys & Girls Club membership.

Table 1 provides the number of children in each condition who participated in~and were retained through! all phases of the study over 24 months. Most of the


results below are reported for 95 children for whom data were collected across all fivewaves from baseline in second grade to follow-up data collection at the beginning offourth grade. ~The number included in analyses varies somewhat for different mea-surement sources.! These include 50 program and 45 control group children. Partici-pants were racially diverse, with approximately 63% Black, 35% Hispanic, and 2%Caucasian. Forty-seven percent of the children were female, and 53% were male.Table 2 illustrates the race and gender composition of the program and controlconditions. All study Clubs and schools were located in economically distressed neigh-borhoods. The percentages of children who qualified for the free and reduced schoollunch program at each site were 98, 78, and 66.

Table 1. Retention in the Study by Condition and Time

Grade 2 Grade 3 Grade 4

Posttest

1 2 3 4

ConditionPretest

(n! n%

Retention n%

Retention n%

Retention n%

Retention

Program 82 76 92.7 67 81.7 64 78.1 57 69.5Control 74 68 91.9 63 85.1 60 81.1 51 68.9Total 156 144 92.3 130 83.3 124 79.5 108 69.2

Children with Complete TASS Data at Each Administration and All Prior AdministrationsProgram 82 76 92.7 66 80.5 58 70.7 50 61.0Control 74 67 90.6 60 81.1 55 74.3 45 60.8Total 156 143 91.7 126 80.8 113 72.4 95 60.9

Note: Because the results we report focus primarily on Teacher Assessments of Student Skills ~TASS!, information reportedhere reflects the number of children with complete TASS data.

Table 2. Race/Ethnicity and Gender by Condition

Program Control

n % n %

Race0EthnicityCaucasian 2 3.9 0 0Hispanic 17 33.3 16 36.4Black 32 62.7 28 63.6Total 51 53.7 44 46.3

GenderMale 29 56.9 21 47.7Female 22 43.1 23 52.3Total 51 53.7 44 46.3

Note: Because the results we report focus primarily on Teacher Assessments ofStudent Skills ~TASS!, the information reported here applies to children withcomplete TASS data.


Measures

Multiple outcome measures were employed, drawing upon different data sources.These included teacher assessments of children, academic records, and self-reportquestionnaires completed by children and parents.

Teacher Assessment of Student Skills (TASS). Over five waves of data collection ~in the falland spring of second- and third-grade and in the fall of fourth grade!, teacherscompleted the TASS measure for each child in the study. The original TASS ~Godin,GoWell, Davis & Kaye, 1990! contains a list of 12 personal and social competence skillson which children were rated by their classroom teachers. The investigators addedfour items, three of which are academically oriented. The possible range of teacherratings is from 1 ~extraordinarily poor! to 7 ~extraordinarily good! for children’sdevelopment of the particular skill or behavior. Table 3 presents the 16 items on thefinal TASS, along with a sample phrase that helped describe the meaning of each itemto teachers.

Academic Records. Children’s quarterly grades were accessed from academic records ateach school. Grades were standardized within school and marking period. This al-lowed us to contrast the randomly assigned program and control groups while con-trolling for possible site, school, and time differences in the assignment of grades.Yearly achievement test scores also were collected. However, efforts to create com-parable test scores proved difficult because of the use of different tests with differentsubscales across sites, and further because of the use of different tests within sitesacross time.

Table 3. Teacher Assessment of Student Skills (TASS) Items and Description

Item Description

Coping with conflict Can express anger with nonaggressive words rather thanusing physical action or aggressive words

Decision making Child thinks before he0she decides to actMaking conversation Pays attention to the person speakingPositive attitude toward others Compliments othersCommunication skills ~assertiveness! Can communicate needs to another child without being

pushy or demandingEthical behavior Tells the truth when asked about possible wrongdoingExpressing feelings Can describe one’s own feelings or mood verballyPositive attitude toward self Is willing and enjoys having one’s work displayedRefusal skills Can say no to a wrongdoing, even when under peer

pressureSelf-care Dresses neatlyClassroom enthusiasm Participates actively in classProblem solving Solves peer problems without help from adultsCourteousnessa Is courteous to teacher and other school personnelLanguage0Artsa Expresses self well in writingMatha Uses number facts accuratelyHomeworka Completes homework assignments

aItems added by the investigators for purposes of this study.


Child Questionnaire. Child surveys were administered in the fall and spring of secondand third grade and in the fall of fourth grade for a total of five waves of datacollection. The self-report questionnaire was designed to assess perceptions of theharmful effects of alcohol, tobacco, and other drugs, bonding to school, and familyfunctioning and management practices. Items previously were field tested to ensurethat they were age-appropriate. More specifically, separate scales on the survey mea-sured the following constructs ~with the number of items in a scale!: intent to usealcohol, tobacco, and other drugs ~4!; perceptions of whether alcohol, tobacco, andother drugs are good0bad ~4!; feelings about school ~i.e., school bonding! ~4!; attitudetoward school ~3!; attitude toward teacher ~8!; perceptions of family cohesiveness ~6!;reported use of physical punishment in the family ~4!.

Parent Questionnaire. A self-report questionnaire was administered to children’s parentsin the fall of 1997 ~pretest! and again in the spring of the second and third grade. Dueto difficulty in accessing parents for completion of the survey, the parent question-naire was not administered subsequently. The questionnaire assessed parents’ familyfunctioning and management practices, the extent to which they supported theirchild in school-related activities, and their attitudes toward alcohol, tobacco, andother drugs. The number of respondents to the parent survey was small, and nosignificant treatment effects were observed; the parent survey is not discussed furtherhere.

Statistical Analyses

The primary method of analysis was the repeated measures analysis of covariance~ANCOVA!, with condition ~program vs. control! as the independent variable, scoresfor the posttests and follow-up assessment as levels of the repeated measures factor~i.e., the dependent variable!, and the pretest score as the covariate ~see Games, 1990,for details!. For instance, for the teacher ratings on the TASS scores for the threeposttests and follow-up ~measured, respectively, at 6, 12, 18, and 24 months after thepretest! served as levels of the repeated measures factor, while pretest ratings served asthe covariate. The individual ANCOVAs were preceded by a multivariate repeated-measures analysis of covariance ~MANCOVA!.

In presenting results, we focus on significant effects of condition and on signifi-cant interactions of condition and time. When a condition 3 time interaction wasobserved, post hoc tests were conducted on the covariance-adjusted means ~Games,1990! to determine the nature of the interaction.

RESULTS

Initial Equivalence and Attrition

To assess the initial equivalency of the two conditions, scores on the pretest measureswere subjected to a 2 3 2 analysis of variance ~ANOVA!, with condition ~program vs.control! and attrition status ~completed vs. attrition! as the independent variables~with completion operationalized in terms of providing data at all measurementoccasions!.

The analyses revealed a marginally significant effect of attrition status for twoTASS variables: Positive Attitude toward Others, F~1,152! 5 3.20, p , .08; and Making


Conversation, F~1,152! 5 3.15, p , .08. Relative to those who failed to complete allfive measurement occasions, those for whom complete data were available were ini-tially rated as having more positive attitudes toward others ~mean 5 4.17 and 4.54, forthe attrition and completion groups, respectively!, and initially rated as having betterconversation skills ~mean 5 4.25 and 4.65, for the attrition and completion groups,respectively!.

A significant main effect of attrition status also was found for children’s standard-ized spelling grades, F~1,135! 5 8.67, p , .005. Children with complete report carddata over the five measurement waves received significantly higher standardized spell-ing grades at pretest ~mean 5 .16! than did those with incomplete report card data~mean 5 2.39!.

Attrition analyses also revealed one significant effect of condition on children’sFeelings about School at pretest, F~1,146! 5 4.88, p , .05. Unexpectedly, programchildren expressed more positive feelings about school at pretest ~mean 5 4.54! thandid control children ~mean 5 4.41!.

Demographic Characteristics. To assess equivalency on race0ethnicity and gender, thenumber and percentage of those who were Caucasian, Black, or Hispanic, and num-bers and percentages of those who were male and female were compared. Table 2shows race0ethnicity and gender by condition for those who completed all five mea-surement occasions. Analyses revealed no condition differences with respect to race0ethnicity, x2~2! 5 1.79, ns, or with respect to gender, x2~1! 5 .79, ns.

Summary. With respect to attrition, those children who completed all five testingoccasions in the study and therefore were included in the analysis did not differ onmost variables from those children who failed to complete the study. Marginallysignificant initial differences were found, however, on teacher ratings of PositiveAttitude toward Others and Conversation skills. Children who remained in the studyreceived initial scores on these measures that were somewhat higher than those receivedby youth who did not complete the study. Children who remained in the study alsoreceived higher initial spelling grades. This could suggest some need for caution whengeneralizing the results. Except for one unexpected initial difference on Feelingsabout School, the program and control groups were initially equivalent on all mea-sures. Importantly, there was no evidence of differential attrition across conditions onthe demographic and outcome variables. This suggests that the benefits of randomassignment remain.

Outcome Variables: Teacher Assessment of Student Skills

A MANCOVA was conducted with the 16 TASS items serving as the dependent vari-ables. The Condition 3 Time interaction was marginally significant, F~48,590! 5 1.33,p 5 .071, indicating that the program and control groups tended to have differentpatterns on the TASS across time. Although this effect did not achieve the conven-tional level of significance ~two tailed!, this marginal effect on the 16 TASS itemsseems to justify further examination of the individual items.

Refusal Skills. Controlling for pretest scores, there was a significant Condition 3 Timeinteraction for teacher ratings of Refusal Skills, F~3,267! 5 3.07, p , .05. As Figure 1illustrates, whereas the Refusal Skill ratings of the program group did not change


significantly across time, the Refusal Skills of control children actually decreasedsignificantly over time. Children in the control condition received significantly lower~i.e., less positive! Refusal Skill ratings in third grade ~Posttests 2 and 3! compared totheir second-grade ratings ~Posttest 1!.

Problem Solving. A significant Condition 3 Time interaction also emerged for ProblemSolving, F~3,261! 5 3.50, p , .05. As shown in Figure 2, teacher ratings of theProblem-Solving skills of program children did not differ significantly across the fourposttests. On the other hand, teachers rated control children as significantly lower onProblem Solving in third grade and in the fall of fourth grade ~Posttests 2, 3, and 4!,relative to their second-grade ratings ~Posttest 1!.

Courteousness. The Condition 3 Time interaction for teacher ratings of Courteousnesswas near conventional levels of significance, F~3,267! 5 2.56, p , .06 ~two tailed!. Asshown in Figure 3, whereas the Courteousness ratings of program children did notdiffer significantly across time, the ratings of control children decreased significantlyat Posttest 2 and then remained fairly stable over subsequent measurements.

Ethical Behavior. The Condition 3 Time interaction for teacher ratings of Ethical Behav-ior also approached conventional levels of significance, F~3,267! 5 2.50, p , .07. Asillustrated in Figure 4, although teacher ratings of Ethical Behavior decreased signif-icantly for children in both conditions, the decrease occurred earlier in the controlgroup ~at Posttest 2! than in the program group ~Posttest 3!. By the fall of fourth grade~Posttest 4!, however, Ethical Behavior ratings increased for both groups to their initiallevels.

Figure 1. Mean refusal skills TASS ratings by Condition and Time, adjusted forpretest ratings.


Summated TASS Scale. Based on the previous findings, we constructed a summatedscale for the previously reported TASS items. Reliabilities across the five measurementwaves range from .89 to .93. A significant Condition 3 Time effect emerged for thissummated teacher rating scale, F~3,261! 5 3.72, p , .05. As with the single TASS

Figure 2. Mean problem-solving TASS ratings by Condition and Time, adjusted forpretest ratings.

Figure 3. Mean courteousness TASS ratings by Condition and Time, adjusted forpretest ratings.


items, scores on the summated scale do not differ for program children across thefour posttests. For the control children, however, scores on the summated scale decreasedsignificantly in the third grade and fall of fourth grade ~Posttests 2, 3, and 4! relativeto second-grade ratings ~Posttest 1!.1

No Difference Findings. No significant differences among the groups were observed forteacher ratings of Coping with Conflict, Decision Making, Making Conversation, Pos-itive Attitude toward Others, Communication Skills, Expressing Feelings, Positive Atti-tude toward Self, Self-Care, Classroom Enthusiasm, Language Arts, Math, or Homework.

Child Survey

The child survey included a four-item scale measuring the child’s Feelings AboutSchool. Children were asked to rate their feelings about various aspects of school ona five-point scale ranging from 1 ~really don’t like! to 5 ~really like!. Illustrative itemsin this scale include: “How I feel about the things we do in school” and “How I feelabout my teacher.”

An ANCOVA, based on five administrations of the child survey, revealed a signif-icant main effect of Condition on Feelings about School, F~1,77! 5 6.08, p , .05.Overall, program children expressed more positive feelings about school ~mean 54.44! than did control group children ~mean 5 4.21!. This effect was qualified, how-

1 For the TASS items for which a significant Condition 3 Time effect occurred, eta squared ~a commonindex of effect size used in ANOVA designs! and the observed power for the Condition 3 Time effect wereas follows: refusal skills, .033., .715; problem solving, .039, .776, courteousness, 0.28, 6.28; ethical behavior,.027, 6.14; and summated TASS scale, .041, .803.

Figure 4. Mean ethical behavior TASS ratings by Condition and Time, adjusted forpretest ratings.


ever, by a significant Condition 3 Site interaction, F~2,77! 5 3.85, p , .05. Follow-uptests of the interaction revealed that program children expressed significantly morepositive feelings toward school than did control children in two out of the three studysites. In the remaining site, program and control children did not differ significantlyin their positive feelings about school. There were no significant effects for othermeasures on the child survey.2

Grades

An ANCOVA was conducted on math, spelling, and reading ~standardized! gradesbased on eight grading periods, with the first grading period serving as a covariate.For spelling grades, there was a significant main effect of Condition, F~1,93! 5 4.67,p , .05. Overall, program children’s standardized spelling grades ~mean 5 .12! werehigher than those of control children ~mean 5 2.10!. This effect occurred in thecontext, however, of a marginally significant Condition 3 Site interaction, F~2,93! 52.78, p , .07. Follow-up tests revealed that program children received significantlyhigher standardized spelling grades than did controls at two out of the three programsites.3 For the remaining site, program and control children did not differ signifi-cantly in their standardized spelling grades. No significant differences were observedon standardized math and reading grades.

DISCUSSION

This study sought to determine the effectiveness of a 2-year multicomponent inter-vention targeting second-grade children, schools, and parents, and carried out in Boys& Girls Clubs in collaboration with schools. Would the intervention reduce risk factorspredictive of future substance abuse and0or enhance protective factors that bufferchildren from risk, relative to the control condition?

Findings from this 2-year longitudinal study indicate that the program had rela-tively positive effects on children’s personal competency skills including ~1! refusingwrongdoing; ~2! solving peer and school problems; ~3! showing courteousness toteachers and other school personnel; and ~4! behaving ethically. For the most part,the program also had positive effects on children’s feelings toward school and gradesin spelling. Specifically, ratings for three of the four personal competency skills ~exceptethical behavior! and overall competency skills held constant for program childrenfrom the beginning of second grade through the fall of fourth grade, while ratings onthese skills for control group children significantly decreased in third grade relative totheir second grade ratings. Control group children also showed a significant decreasein problem solving at the fourth-grade follow-up. At two of the three study sites,program children expressed more positive feelings about school and earned higherspelling grades than control group children.

These results indicate that the intervention provided program children with pro-tective factors that, for the most part, were sustained until fourth grade, months afterthe 2-year program was completed. Our findings are consistent with previous research

2 The eta squared and observed power were .073 and .683, respectively, for the main effect of Condition onFeelings about School, and .091 and 6.82 for the Condition 3 Site interaction.3 The eta squared and observed power were .048 and .571, respectively, for the main effect of Condition onSpelling Grades, and 0.56 and 5.35, respectively, for the Condition 3 Site effect.


~e.g., Elias & Clabby, 1992; Schinke & Tepavac, 1995!, showing that social competencypromotion interventions for elementary school children can improve behavioral problem-solving skills, relationships with peers, and behavior. The intervention apparentlyprotected program children from a trajectory of decreasing competency skills thattheir peers experienced as they progressed from second grade to the beginning offourth grade. These findings are particularly noteworthy because these prosocial skillshave been shown to protect elementary school children from behavior problemsassociated with increased risk of future substance abuse ~Hawkins et al., 1992!.

The program had limited effects on children’s academic performance. However,this finding is somewhat less disappointing given that social adjustment in the earlyelementary grades has been found to be more important than academic performancein preventing future substance abuse ~e.g., Jessor, 1976; Robins, 1984!. Moreover,teachers’ positive ratings of children in their classrooms indicated their approval ofprogram students’ behaviors. According to Comer ~1988!, when social skills are con-sidered appropriate by the teacher, they elicit positive responses that develop a bondbetween the child and teacher. This relationship contributes to the child’s overalldevelopment, and helps protect the child from future substance abuse. Furthermore,positive feelings toward school reported by program children at two of the three studysites provides additional evidence that the program may have contributed to schoolbonding and social adjustment in school.

In considering which program component~s! produced these effects, there isreason to believe that the SMART Kids after-school program had the major influencein combination with the SMART Teachers component. Parent involvement in theSMART Parents program was minimal and inconsistent, in part because parent involve-ment is not simple to elicit, especially in at-risk communities, and in part because theprevention coordinators were extremely busy planning and running the daily after-school program. Prevention coordinators were not able to invest the time and effortrequired to develop the necessary trust and ongoing relationships necessary to engageand sustain involvement by high-risk parents ~St. Pierre et al., 1997; St. Pierre &Kaltreider, 1997!.

It appears that the overall SMART Kids after-school program at the Boys & GirlsClubs contributed to these positive program effects. Although the 2-year small-groupprevention curriculum focused on enhancing children’s personal and social compe-tence skills, the overall after-school program may have provided other protectivefactors. In everyday activities of homework assistance, tutoring, recreational activities,and even snack time, program staff attempted to promote values of caring, responsi-bility, and learning. Program staff were warm and supportive, and they tried to createa caring Boys & Girls Club environment that emphasized positive social norms, pro-social behavior, and educational success, factors shown to protect or mitigate againstrisk for later antisocial behavior and substance abuse ~Brewer, Hawkins, Catalano, &Neckerman, 1995; Rutter, Maughan, Mortimore, Ouston, & Smith, 1979!.

It also appears that the Boys & Girls Clubs collaborations with local schools weresuccessful in their overall goal to work together to carry out the program. Preventioncoordinators systematically met with children’s classroom teachers every month todiscuss ways the program could meet individual children’s social and academic needs.Teachers also encouraged program children to attend the after-school program whenthe school day ended. Between monthly meetings, teachers contacted preventioncoordinators when problems arose with program children. Prevention coordinatorsand teachers worked together to address those issues. During our site visits, several


teachers and principals acknowledged the positive relationships the program hadcreated with the Boys & Girls Clubs and expressed their appreciation for the program.

Positive findings from this study, though primarily for personal competency skills,and to a lesser degree for feelings toward school, indicate that the collaborative Boys& Girls Club and school intervention provided a number of protective factors thatmay buffer children from negative effects of exposure to the multiple risks in theirlives for future drug abuse. These positive findings are very encouraging, given theirsalience for early elementary school-age children. Moreover, a major strength of thestudy, and a relative rarity in intervention studies with elementary school-age chil-dren, is the use of an experimental design with random assignment of individualchildren to conditions. Analyses of initial equivalence of program and control chil-dren on all measures verified that the random assignment was successful in creatingequivalent groups. This initial equivalence provides a level of confidence in thefindings that does not exist with more typical quasi-experimental designs that areoften plagued by selection bias.

Findings from this study illustrate the potential for youth-serving organizations tocollaborate with local schools to implement multicomponent prevention programs,especially for younger elementary school children living in low-income neighbor-hoods. However, given the rarity of empirical studies of such programs for childrenthis age ~Coie et al., 1993! and the importance of implementing programs early indevelopment, more rigorous studies are needed to determine the efficacy of variousforms of multicomponent programs for different ages of high-risk elementary schoolchildren.

ACKNOWLEDGMENTS

This research was funded by a grant from the Center for Substance Abuse Prevention,Substance Abuse and Mental Health Services Administration, Department of Healthand Human Services, Grant No. H1SPO6520A.

REFERENCES

Benard, B., Fafoglia, B., & Perone, J. ~1987, February!. Knowing what to do—and not todo—reinvigorates drug education. ASCD Curriculum Update. Alexandria, VA: ASCD.

Benson, P.L. ~1997!. All kids are our kids: What communities must do to raise caring andresponsible children and adolescents. San Francisco: Jossey-Bass.

Benson, P.L., Scales, P.C., & Roehlkepartain, E.C. ~1999!. A fragile foundation: The state ofdevelopmental assets among American youth. Minneapolis: Search Institute.

Bien, N.Z., & Bry, B.H. ~1980!. An experimentally designed comparison of four intensities ofschool-based prevention programs for adolescents with adjustment problems. Journal ofCommunity Psychology, 8, 110–116.

Blechman, E.A., Taylor, C.J. & Schrader, S.M. ~1981!. Family problem solving versus home notesas early intervention with high-risk children. Journal of Consulting and Clinical Psychology,49, 919–926.

Brewer, D.D., Hawkins, J.D., Catalano, R., & Neckerman, H. ~1995!. Preventing serious, violent,and chronic juvenile offending: A review of evaluations of selected strategies in childhood,adolescence, and the community. In J.C. Howell, B. Krisberg, J.D. Hawkins, & J.H. Wilson~Eds.!, Sourcebook on serious, violent, and chronic juvenile offenders ~pp. 61–141!. Thou-sand Oaks, CA: Sage Publications.


Brook, J.S., Brook, D.W., Gordon, A.S., Whiteman, R., & Cohen, P. ~1990!. The psychosocialetiology of adolescent drug use: A family interactional approach. Genetic, Social and Gen-eral Psychology Monograph, 116, 111–267.

Brook, J.S., Gordon, A.S., Whiteman, M., & Cohen P. ~1986!. Some models and mechanisms forexplaining the impact of maternal and adolescent characteristics on adolescent stage ofdrug use. Developmental Psychology, 22, 460–467.

Brooks, L.S., Linkoff, I.F., & Whiteman, M. ~1977!. Peer, family, and personality domains asrelated to adolescents’ drug behavior. Psychological Reports, 41, 1095–1102.

Coie, J.D., Watt, N.F., West, S.G., Hawkins, J.D., Asarnow, J.R., Markman, H.J., Ramey, S.L.,Shure, M.B., & Long, B. ~1993!. The science of prevention: A conceptual framework andsome directions for a national research program. American Psychologist, 48~10!, 1013–1022.

Comer, J.P. ~1988!. Educating poor minority children. Scientific American, 259, 42–48.Coombs, R.H., & Landsverk, J. ~1988!. Parenting styles and substance use during childhood and

adolescence. Journal of Marriage and the Family, 50, 473–482.Davis, K., & GoWell, D. ~1990!. The million dollar machine. Mt. Holly, NJ: GoWell and Kent, Inc.Dryfoos, J.G. ~1998!. Safe passage: Making it through adolescence in a risky society. New York:

Oxford University Press.Elias, M.J., & Clabby, J.F. ~1989!. Social decision-making skills: A curriculum guide for the

elementary grades. Rockville, MD: Aspen.Elias, M.J., & Clabby, J.F. ~1992!. Building social problem-solving skills: Guidelines from a

school-based program. San Francisco: Jossey-Bass.Elliott, D.S., & Voss, H.L. ~1974!. Delinquency and dropout. Lexington: Heath and Company.Feldhusen, J.F., Thurston, J.R., & Benning, J.J. ~1973!. A longitudinal study of delinquency and

other aspects of children’s behavior. International Journal of Criminology and Penology, 1,341–351.

Friedman, A.S. ~1983, July!. High school drug abuse clients. In Clinical research notes. Rock-ville, MD: Division of Clinical Research, National Institute on Drug Abuse.

Games, P.A. ~1990!. Alternative analyses of reported measures designs by ANOVA and MAN-COVA. In A. Von Eye ~Ed.!, Statistical methods in longitudinal research ~vol. 1, pp. 81–121!.New York: Academic Press.

Garmezy, N. ~1985!. Stress-resistant children: The search for protective factors. In J.E. Stevenson~Ed.!, Recent research in developmental psychopathology. Journal of Child Psychology andPsychiatry, 4~Suppl.!, 213–233.

Godin, S., GoWell, D., Davis, K., & Kaye, E. ~1990, November!. Teachers’ ratings of elementaryschool students life skills: Impact of a model drug abuse prevention program. Paper pre-sented at the annual meeting of the New Jersey Psychological Association, Summerset, NJ.

Harachi, T.W., Ayers, C.D., Hawkins, J.D., Catalano, R.F., & Cushing, J. ~1995!. Empoweringcommunities to prevent adolescent substance abuse: Process evaluation results from a risk-and protection-focused community mobilization effort. Journal of Primary Prevention, 16,233–254.

Hawkins, J.D., Arthur, M.W., & Catalano, R.F. ~1995!. Preventing substance abuse. In M. Tonry& D. Farrington ~Eds.!, Crime and justice: A review of research: vol. 19. Building a safersociety: Strategic approaches to crime prevention. Chicago: University of Chicago Press.

Hawkins, J.D., Catalano, R.F., & Miller, J.Y. ~1992!. Risk and protective factors for alcohol andother drug problems in adolescence and early adulthood: Implications for substance abuseprevention. Psychological Bulletin, 112, 64–105.

Hawkins, J.D., Kosterman, R., Maquin, E., Catalano, R., & Arthur, M.W. ~1997!. Substance useand abuse. In R.T. Ammerman & M. Hersen ~Eds.!, Handbook of prevention and treatmentwith children and adolescents ~pp. 203–237!. New York: John Wiley & Sons, Inc.


Howell, J.C., Krisberg, B., Hawkins, J.D., & Wilson, J.J. ~Eds.! ~1995!. A sourcebook: Serious,violent, and chronic juvenile offenders. Thousand Oaks, CA: Sage Publications.

Institute of Medicine ~IOM!, Committee on Prevention of Mental Disorders. ~1994!. In P.J.Mrazek & R.J. Haggerty ~Eds.!, Reducing risks for mental disorders: Frontiers for preventiveintervention research. Washington, DC: National Academy Press.

Jessor, R. ~1976!. Predicting time of onset of marijuana use: A developmental study of highschool youth. Journal of Consulting and Clinical Psychology, 44, 125–134.

Jessor, R., & Jessor, S.L. ~1977!. Problem behavior and psychosocial development: A longitudinalstudy of youth. New York: Academic Press.

Kaltreider, D.L., & St. Pierre, T.L. ~1995!. Beyond the schools: Strategies for implementingsuccessful drug prevention programs in community youth-serving organizations. Journal ofDrug Education, 25~3!, 223–232.

Kandel, D.B., & Andrews, K. ~1987!. Processes of adolescent socialization by parents and peers.International Journal of the Addictions, 22, 319–342.

Kandel, D.B., Kessler, R.C., & Margulies, R.S. ~1978!. Antecedents of adolescent initiation intostages of drug use: A developmental analysis. Journal of Youth and Adolescence, 7, 13–40.

Krosnick, J.A., & Judd, C.M. ~1982!. Transitions in social influence at adolescence: Who inducescigarette smoking? Developmental Psychology, 18, 359–368.

The David and Lucile Packard Foundation. ~1999!. The future of our children: When school isout: Executive summary ~vol. 9!. Los Altos, CA: Author.

Pentz, M.A., Dwyer, J.H., MacKinnon, D.P., Flay, B.R., Hansen, W.B., Wang, E.Y.I., & Johnson,C.A. ~1989!. A multi-community trial for prevention of adolescent drug abuse: Effects ondrug use prevalence. Journal of the American Medical Association, 262, 3259–3266.

Perry, C.L. ~1986!. Community-wide health promotion and drug abuse prevention. Journal ofSchool Health, 56~9!, 359–363.

Robins, L.N. ~1984!. The natural history of adolescent drug use. American Journal of PublicHealth, 74, 656–657.

Rolf, J., Masten, A.S., Cichetti, D., Neuchterlein, K.H., & Weintraub, S. ~Eds.!. ~1990!. Risk and pro-tective factors in the development of psychopathology. New York: Cambridge University Press.

Rutter, M. ~1985!. Resilience in the face of adversity: Protective factors in resistance to psychi-atric disorders. British Journal of Psychiatry, 147, 598–611.

Rutter, M., Maughan, B., Mortimore, P., Ouston, J., & Smith, A. ~1979!. Fifteen thousand hours:Secondary schools and their effects on children. Cambridge, MA: Harvard University Press.

Schinke, S.P., & Tepavac, L. ~1995!. Substance abuse prevention among elementary schoolstudents. Drugs & Society, 8~304!, 15–27.

Smith, G.M., & Fogg, C.P. ~1978!. Psychological predictors of early use, late use, and non-use ofmarijuana among teenage students. In D.B. Kandel ~Ed.!, Longitudinal research on druguse: Empirical findings and methodological issues ~pp. 101–113!. Washington, DC:Hemisphere-Wiley.

St. Pierre, T.L., Kaltreider, D.L., Mark, M.M., & Aikin, K.J. ~1992!. Drug prevention in a com-munity setting: A longitudinal study of the relative effectiveness of a three-year primaryprevention program in Boys & Girls Clubs across the nation. American Journal of Com-munity Psychology, 20~6!, 673–706.

St. Pierre, T.L., Mark, M.M., Kaltreider, D.L., & Aikin, K.J. ~1997!. Involving parents of high riskyouth in drug prevention: A three-year longitudinal study in Boys & Girls Clubs. Journal ofEarly Adolescence, 17~1!, 21–50.

St. Pierre, T.L., & Kaltreider, D.L. ~1997!. Strategies for involving parents of high-risk youth indrug prevention: A three-year longitudinal study in Boys & Girls Clubs. Journal of Com-munity Psychology, 25~5!, 473–485.


Task Force on Youth Development and Community Programs. ~1992!. A matter of time: Riskand opportunity in the nonschool hours. New York: Carnegie Council on AdolescentDevelopment, Carnegie Corporation.

Weissberg, R.P., Gesten, E.L., Carnike, C.L., Toro, P.A, Rapkin, B.D., Davidson, E., & Cowen,E.L. ~1981!. Social problem solving skills training: A competence-building intervention withsecond- to fourth-grade children. American Journal of Community Psychology, 9, 411–423.

Werner, E.E., & Smith, R.S. ~1982!. Vulnerable but invincible: A longitudinal study of resilientchildren and youth. New York: McGraw-Hill.

Werner, E.E., & Smith, R.S. ~1992!. Overcoming the odds: High risk children from birth toadulthood. Ithaca, NY: Cornell University Press.


boys & girls clubs and school collaborations: a longitudinal study of a multicomponent substance...

Documents