bridging the gap between laboratory and clinic in...

Journal of Consulting and Clinical Psychology1995, Vol. 63, No. 5, 688-701

Copyright 1995 by the American Psychological Association, Inc.0022-006X/95/S3.00

Bridging the Gap Between Laboratory and Clinic in Child andAdolescent Psychotherapy

John R. Weisz, Geri R. Donenberg,and Susan S. Han

University of California, Los Angeles

Bahr WeissVanderbilt University

Meta-analyses of laboratory outcome studies reveal beneficial effects of psychotherapy with childrenand adolescents. However, the research therapy in most of those lab studies differs from everydayclinic therapy in several ways, and the 9 studies of clinic therapy the authors have found show mark-edly poorer outcomes than research therapy studies. These findings suggest a need to bridge thelong-standing gap between outcome researchers and clinicians. Three kinds of bridging research areproposed and illustrated: (a) enriching the research data base on treatment effects by practitioners inclinical settings—including private practice and health maintenance organizations, (b) identifyingfeatures of research therapy that account for positive outcomes and applying those features to clinicalpractice, and (c) exporting lab-tested treatments to clinics and assessing their effects with referredyouths. If these bridging strategies were widely adopted, despite the numerous obstacles describedherein, real progress might be made toward more effective treatment in clinical practice.

The gulf that divides clinical practice and clinical research isnow accepted as a fact of life by many in the mental healthprofessions and in academia. For years, practicing clinicianshave maintained that psychotherapy research is of little valueto them (see Elliott, 1983; Kupfersmid, 1988; Luborsky, 1972;Orlinsky & Howard, 1978; Parloff, 1980;Strupp, 1989). Whenclinical psychologists are asked to rank the usefulness to theirpractice of various sources of information, research articles andbooks typically fall near the bottom of the scale (see L. Cohen,1979; Cohen, Sargent, & Sechrest, 1986; Morrow-Bradley &Elliot, 1986). An experienced psychotherapist, S. D. Raw re-cently noted,

Over more than 22 years of clinical practice, I've become increas-ingly disaffected about the value of psychotherapy outcome re-search for the practice of psychotherapy . . . I suspect that out-come research tells us something; I'm just not sure what. (1993,pp. 75-76)

The problem is apparently not confined to long-time practition-ers. A recent series of articles in "The Scientist-Practitioner"

John R. Weisz, Geri R. Donenberg, and Susan S. Han, Department ofPsychology, University of California, Los Angeles (UCLA); Bahr Weiss,Department of Psychology and Human Development, VanderbiltUniversity.

Preparation of this article and conduct of the research reported herewere facilitated by a Research Scientist Award (K.05 MH01161) andresearch grants (RO1 MH38240 and RO1 MH49522) from the NationalInstitute of Mental Health.

We appreciate the assistance provided by Danika Kauneckis.Geri R. Donenberg is now at the Department of Psychiatry, North-

western University.Correspondence concerning this article should be addressed to John

R. Weisz, Department of Psychology, Franz Hall, UCLA, 405 HilgardAvenue, Los Angeles, California 90024-1563.

section of The Behavior Therapist (see Suinn, 1993) has fo-cused on the fact that so many new graduates of our scientist-practitioner PhD programs drift away from journal reading andresearch values when they enter clinical practice. Clearly, di-verse observers from diverse perspectives perceive a marked sep-aration between what goes on in the clinical research laboratoryand what goes on in the practicing clinician's office, and manyquestion the relevance of research to the work of thepractitioner.

In this article, we address the gap between clinical practiceand the research laboratory. We focus on the issue as it relatesspecifically to interventions for children and adolescents(herein referred to as "children"); although we refer frequentlyto "clinics" and much of the evidence we cite is from researchin community clinics, the issues raised may be relevant to thework of clinicians in other settings, including private practiceoffices and health maintenance organizations (HMOs; dis-cussed later). These possibilities warrant research in the future.

In what follows, we suggest (a) how the evidence on therapyoutcomes reveals a need to bridge the gap between lab andclinic, (b) what kinds of bridging research should be on theagenda at this point, and (c) what obstacles may be confrontedalong the way.

Can Outcome Research Help Clinicians?

Perhaps the most important initial question to consider iswhether clinicians might actually profit from the findings ofoutcome research. We suspect that the answer is yes. To explainwhy, we describe the overall picture of evidence on child psy-chotherapy outcomes.

Laboratory Evidence on Child Psychotherapy Effects

Experimental evidence on child and adolescent psychotherapyeffects is perhaps best summarized in the form of meta-analytic

688

SPECIAL SECTION: BRIDGING THE LAB-CLINIC GAP 689

COHEN'S EFFECTSTANDARD SIZE PERCENTILE

1.00nqo

LARGE — - 0.80u./u0.60

MEDIUM o.so

SM;

0.400.30

0.1U000

- 0.10- 0.20

8482797673696662585450 —4642 ]

/ \

BENEFICIALEFFECT

/"

-INO EF\

\

FECTlX

DETRIMENTALEFFECT

Figure 1. An aid to interpreting effect size (ES) statistics. Each ESvalue can be thought of as reflecting a corresponding percentile value(i.e., the percentile standing of the average treated child, after treatment,averaging across outcome measures, relative to the untreated group).

reviews (see Mann, 1990; Smith, Glass, & Miller, 1980; see alsocritiques of meta-analysis, e.g., by Wilson, 1985).' The unit ofanalysis in the meta-analyses we will consider here is the effect size(ES) statistic, an index of the magnitude and direction of therapyeffects. For the studies we consider here, ES is the posttreatmentscore of a treated group on some outcome measure minus the cor-responding score of an untreated or minimally treated controlgroup, with the difference divided by the standard deviation of theoutcome measure. In most meta-analyses, a mean ES is computedfor each study (or each treatment group) by averaging across thevarious outcome measures. Figure 1 provides a guide to interpret-ing the ES values we discuss here (note that the range of possibleES values is much broader than that shown in Figure 1). As Figure1 indicates, ES values may be positive, indicating beneficial treat-ment effects, or negative, indicating detrimental treatment effects.Each ES can be thought of as corresponding to a percentile stand-ing of the average treated child on the outcome measure (ormeasures) if that child were placed in the control group after treat-ment; for example, an ES of 0.50 would indicate that the averagetreated child scored better after treatment than 69% of the controlgroup. Finally, note that Cohen (1992) considers an ES of 0.20to indicate a small effect, 0.50 a medium effect, and 0.80 a largeeffect.

Broad-Based Meta-Analyses of Child PsychotherapyResearch

Four known broad-based child therapy meta-analyses focuson a variety of treated problems and types of intervention.2

Casey and Berman (1985) surveyed outcome studies publishedbetween 1952 and 1983 and focused on children aged 12 andyounger. Mean ES was 0.71 for the studies that included treat-ment-control comparisons; the average treated child scoredbetter after treatment than 76% of control-group children, aver-aging across outcome measures. Weisz, Weiss, Alicke, and Klotz

(1987) reviewed outcome studies published between 1952 and1983 and including children aged 4-18. The mean ES was 0.79;after treatment, the average treated child was at the 79th per-centile of control group peers. Kazdin, Bass, Ayers, and Rodgers(1990) surveyed studies published between 1970 and 1988 in-cluding youngsters aged 4 to 18. For studies that comparedtreatment groups and no-treatment control groups, the meanES was 0.88, indicating that the average treated child was betteroff after treatment than 81% of the no-treatment youngsters.For studies in the Kazdin et al. (1990) collection that involvedtreatment groups vs. active control groups, the mean ES was0.77, indicating that after treatment the average treated childwas functioning better than 78% of the control group. Finally, arecent meta-analysis by Weisz, Weiss, Han, Granger, and Mor-ton (1995), included studies published between 1967 and 1993,and involved children aged 2 to 18. The mean ES was 0.71,indicating that, after treatment, the average treated child wasfunctioning better than 76% of control group children.3 Formore detailed descriptions of the procedures and findings of thevarious meta-analyses, see Weisz & Weiss, 1993.

The evidence from these four broad-based meta-analysesshows rather consistent positive treatment effects; ES valuesranged from 0.71 to 0.84 (estimated overall mean for Kazdin etal., 1990), near Cohen's (1988) threshold of 0.80 for a largeeffect. Figure 2 summarizes findings from the four child meta-analyses and compares them to findings from two often-citedmeta-analyses—Smith and Glass's (1977) analysis of predomi-nantly adult psychotherapy outcome studies, and Shapiro andShapiro's (1982) analysis of exclusively adult psychotherapyoutcome studies. As Figure 2 suggests, mean effects found inchild meta-analyses fall roughly within the range of the meaneffects found in these two adult meta-analyses.

Limitations of the Evidence: Research Therapy VersusClinic Therapy

The meta-analyses cited earlier point to positive effects ofchild psychotherapy. However, as noted elsewhere (e.g., Weisz

' For a more detailed account of studies and meta-analyses reviewedhere, see the book Effects of Psychotherapy With Children and Adoles-cents (Weisz & Weiss, 1993).

2 Six more narrowly focused meta-analyses add to the body of avail-able information on child psychotherapy effects. Baer and Nietzel(1991) reviewed outcome studies involving cognitive and behavioraltreatment of child impulsivity. Durlak, Fuhrman, and Lampman(1991) surveyed outcome studies involving cognitive-behavioral ther-apy for children, across an array of treated problems. Dush, Hirt, andSchroeder (1989) reviewed child outcome studies involving use of thecognitive-behavioral technique of self-statement modification. Hazel-rigg, Cooper, and Borduin (1987) surveyed studies of the outcome offamily therapy. Prout and DeMartino (1986) reviewed studies ofschool-based psychotherapy. Russell, Greenwald, and Shirk (1991) ex-plored the impact of child psychotherapy (of various types) on chil-dren's language proficiency. All six meta-analyses showed positive psy-chotherapy effects overall.

3 The three earliest broad-band meta-analyses each involved tradi-tional analyses without correction for heterogeneity of variance. Weisz,Weiss, et al. (in press) used both traditional analyses and the weightedleast squares (WLS) method recommended by Hedges and Olkin(1985) to control for heterogeneity; traditional unweighted leastsquares (ULS) methods were found to generate higher mean ES thanthe WLS method (means, 0.71 and 0.54, respectively).

690 WEISZ, DONENBERG, HAN, AND WEISS

Figure 2. Mean effect sizes found in the predominantly adult meta-analysis by Smith and Glass (1977),in the exclusively adult meta-analysis by Shapiro and Shapiro (1982), and in four broad-based meta-analyses of psychotherapy outcome studies with children and adolescents.

& Weiss, 1993; Weisz, Weiss, & Donenberg, 1992), the 300-plusstudies included in the child meta-analyses constitute only partof the evidence needed on child psychotherapy outcome. Mostof the studies included in the meta-analyses (particularly therecent and behavioral studies) appear to have involved children,interventions, or treatment conditions that are not very repre-sentative of conventional clinical practice. In many of thesestudies, (a) youngsters were recruited for treatment and werenot actual clinic cases; (b) samples were selected for homoge-neity, with therapy addressing one or two focal problems (e.g.,a specific phobia); (c) therapists received concentrated prether-apy training in the specific intervention techniques they woulduse; or (d) the therapy involved primary or exclusive adherenceto those specific techniques. In addition, (e) therapy was oftenhighly structured, frequently guided by a manual or monitoredfor its adherence to a treatment plan.

These features of the experimental studies tend to coalescearound a genre that we refer to here as research therapy, a genrethat differs in a number of ways from conventional clinic ther-apy. Table 1 summarizes some of the most common differencesbetween the two genres. Certainly no single feature listed in Ta-ble 1 under Research Therapy is present in all laboratory out-come studies, nor is any single feature listed under Clinic Ther-apy present in all clinic-based treatment. However, differencesbetween therapy in clinics and therapy in outcome studies arecommon enough that one can certainly understand how clini-cians might question the relevance of most outcome research to

their own clinical work. Beyond this issue, an important ques-tion arises as to whether the positive outcomes that have beendemonstrated in the research therapy studies, and summarizedin the aforementioned meta-analyses, are representative of theoutcomes achieved through actual clinical practice withchildren.

Evidence on the Effects of Clinic Therapy

To address this question, we carried out a search for outcomestudies focused on what might fairly be called clinic therapy.Because a lack of commonly used key words hampered a com-puter-based search, we hand-searched 23 journals known topublish treatment outcome studies; we searched the 2 decadesfrom 1972 through 1991, adding studies identified through ref-erence trails and other sources (e.g., review articles). We aimedfor studies that involved (a) treatment of clinic-referred (notrecruited or "analog") youngsters, (b) treatment in service-ori-ented clinics or clinical agencies, not in research settings (e.g.,not a university lab or a school), (c) therapy conducted by prac-ticing clinicians (as opposed to, e.g., research assistants), and(d) therapy conducted as a part of the regular service-relatedprogram of the clinic, not primarily for research. We requiredthat the studies involve direct comparison between youngsterswho received treatment and a control group who received noneor a placebo condition. This ruled out, for example, treatmentin hospital and residential settings where control groups had


Table 1Some Modal Characteristics of Research Therapy and Clinic Therapy

Research therapy Clinic therapy

Recruited patients-clients (less severe, studyvolunteers)

Homogeneous groupsNarrow problem focusTreatment in lab or school settingsTreated by researchers-assistantsSmall therapist caseloadsSpecial pretherapy preparationShort duration, fixed number of sessionsSingle, focused treatment methodBehavioral interventionsPreplanned, highly structured (treatment

manual, close monitoring of therapistadherence)

Clinic-referred patients-clients (more severe, somecoerced)

Heterogenous groupsBroad, multiproblem focusTreatment in clinics or hospitalsTreated by cliniciansLarge therapist caseloadsLittle pretherapy preparationLonger duration, fewer session limitsMultimethod, eclecticNonbehavioral interventionsRexible, adjustable (no treatment manual, less

monitoring of therapist behavior)

some intervention, although we discuss such studies later in thisarticle. Also excluded were studies involving only parent treat-ment or teacher training and studies focused on children beingtreated for medical conditions.

The search taught us an important lesson: Clinic studies thatmeet the aforementioned criteria are very rare. We found onlynine studies that seemed to fit, and most of these had been pub-lished years ago. The studies all compared treatment and con-trol groups, but by means of several different methodologies.Two studies (DeFries, Jenkins, & Williams, 1964; Smyrnios &Kirkby, 1993) involved random assignment of children to treat-ment and no-treatment groups; five studies compared treatedchildren to clinic dropouts (Ashcraft, 1971; Jacob, Magnussen,& Kemler, 1972; Lehrman, Sirluck, Black, & Glick, 1949; Lev-itt, Beiser, & Robertson, 1959; Weisz & Weiss, 1989); one studycompared treated children to children who were seen for evalu-ation only (Witmer & Keller, 1942); and one study comparedtreated children to demographically and clinically matchedchildren from the general population (Shepherd, Oppenheim,& Mitchell, 1966). (For a discussion of problems and issues instructuring treatment-control comparisons for clinic studies,see Weisz & Weiss, 1989,1993.)

To summarize findings of these nine studies for comparisonwith the meta-analytic evidence, we computed, for each of thenine studies, an ES or ES estimate (for reports that did not in-clude the statistics needed for standard ES calculation, we usedestimation procedures described by Smith et al., 1980, andGlass, McGaw, & Smith, 1981). For studies reporting resultswith more than one outcome measure or more than one pointof outcome assessment (e.g., 1-year and 2-year follow-ups), wecomputed separate ES values for each measure or each assess-ment point and then computed the mean of these values to gen-erate a single ES for each treatment-control group comparisonin each study. As reflected in Figure 3, the ES values rangedfrom —0.40 to 0.29, with the mean ES of the nine clinic studies(0.01) well below the mean ES of the four broad-band meta-analyses discussed earlier (0.77). The evidence thus far suggeststhat the outcomes of clinic therapy may be less positive than theoutcomes of research therapy.

Implications (and Limitations) of the Clinic Findings

If the findings of these clinic studies are taken at face value,they suggest that, for clinic-referred children treated by practic-ing clinicians in clinic settings, the impact of psychotherapymay not be as positive as the impact of research therapy in theexperimental outcome studies. This, in turn, might suggest thatexperimental outcome research has a good deal to offer thepracticing clinician—most importantly, an array of interven-tions that have been shown to produce positive effects. On theother hand, the nine clinic studies we found represent a smallbase of information, much of it quite dated. Clearly, more out-come research on clinic therapy—research that meets the beststandards of methodological rigor and reflects therapy as cur-rently practiced in clinics—is needed. This brings us to the firstelement of what we believe would be a useful research agendalinking lab and clinic.

/. Enriching the Data Base on Outcomes of TraditionalClinic Therapy

The meager payoff generated by our search for clinic-basedoutcome studies reveals how thin the base of published infor-mation is on outcomes for children who receive traditional ther-apy in practice settings. This is particularly surprising given thescientist-practitioner values our discipline has formally es-poused. Those values would seem to support ongoing efforts byclinics and by individual practitioners to collect information onthe youngsters they treat, before and after treatment. Such in-formation could help link the practice and research communi-ties, to the benefit of both, by facilitating comparison of changesin youths treated in lab research versus clinical practice.

The comparisons of randomly assigned treatment and con-trol groups seen in laboratory research are difficult to carry outin practice settings, in part because of the ethical and publicconcerns that arise; but such designs, particularly if they involvewaiting-list or alternate treatment control groups, may be feasi-ble in some clinics. In cases in which no random assignment ispossible, single-case research designs and the use of multiplebaseline procedures may be quite informative (see Kazdin,


CLINIC STUDY EFFECT SIZES

1.0-0.9-

LARGE 0.8

SMALL 0.2

ASHCRAFT DEFRIES(1871) ETAL.

(1964)

JACOB LEHRMAN LEVITT SHEPHARD SMYRNIOS WEISZ WITMERETAL ETAL ETAL. ETAL. 4 KJRKBY & WEISS S KELLER(1972) (1949) (1966) (1966) (1993) (1989) (1942)

Figure 3. Estimated effect sizes for nine studies of clinic-based psychotherapywith children and adolescents.

1992, 1994). In settings in which even these approaches arenot considered workable, useful analytic strategies have beendeveloped for application to pre- and posttreatment data fromclinics in which no aspect of intake or treatment has been al-tered for research purposes (for details plus a discussion of ad-vantages and limitations, see Weisz & Weiss [1989; 1993, espe-cially pp. 70-89] and Weisz et al. [1992]); although such strat-egies require thorough testing of alternative interpretations,they can provide useful information bearing on the effects ofconventional clinical practice with referred youths. Becausedata collection and analysis for such research can be complex,time consuming, and expensive, work of this sort may requirean active partnership between clinicians and researchers—col-laboration that might prove informative for both groups.

An alternative approach to enriching our data base is to testthe impact of traditionally clinic-based treatments under in-creasingly controlled conditions. Such an approach, in princi-ple, offers opportunities to separate effects of traditional treat-ments themselves from effects of the conditions under whichthey are most often carried out in clinics. One example of thisapproach is the Vanderbilt School-Based Counseling Program(Catron & Weiss, 1994), in which licensed mental health prac-

titioners use their own preferred methods of therapy but treatchildren identified through a research protocol, under moni-tored conditions, in schools rather than clinics.

Beyond the rather broad-based approaches noted earlier,there could be real informational value in addressing within-clinic empirical questions—for instance, testing whether cur-rent methods are associated with more positive change in somegroups of youths than in others (as defined, e.g., by age, gender,or treated problems)—assessing whether children treated bycertain therapists or with certain treatment models have betteroutcomes than others, and identifying factors that predict out-come. Such within-clinic analyses could be compared withcounterparts in the laboratory research literature, and thesecomparisons could help in discovering the extent to which pat-terns found in controlled research are representative of patternsseen in clinical settings.

Another important objective for research is to answer a ques-tion that remains virtually unaddressed, to our knowledge:What are the effects of child psychotherapy in such commontreatment configurations as individual and group private prac-tice and HMOs? Certainly, evidence on the child outcomes as-sociated with these forms of practice is needed if scientists and


practitioners are to know how outcomes across the range of to-day's clinic therapies compare with the outcomes achievedthrough laboratory interventions. Generating such informationseems a particularly important enterprise, given certain appar-ent similarities between community clinic-based therapy andprivate practice-HMO-based therapy. Although the latter maybe associated with a more affluent clientele and thus with fami-lies who have access to more resources, it would not be surpris-ing to find that both clinic-based practice and private-HMO-based practice tend to involve features seen in the right-handcolumn of Table 1, features such as (a) clients disturbed enoughto have been referred for treatment (i.e., not recruited), (b)rather heterogeneous groups of clients, (c) rather broad, multi-problem focus, (d) treatment in clinical settings (e.g., not inschools), (e) treatment provided by clinicians (i.e., not bytrained research assistants), ( f ) rather large therapist caseloads,(g) relatively modest pretherapy preparation for individualcases, (h) relatively multimethod, eclectic therapist orienta-tions, (i) most often nonbehavioral treatment methods, and (j)relatively flexible, nonmanualized procedures.

Although it is by no means certain, it is possible that furtherresearch on child outcomes after therapy in various practice set-tings would reveal therapy effects more modest than those foundin laboratory studies. Such an outcome might be disappointing,but it would not need to be seen as entirely discouraging. Cer-tainly it is very good news that the numerous research therapystudies summarized in four meta-analyses indicate that underthe right conditions child therapy can have substantial positiveeffects. If therapy effects in clinical practice prove to be modest,perhaps those effects can be improved by drawing lessons fromthe success of research therapy. To identify the appropriate les-sons, however, researchers need to identify those features of re-search therapy that are most responsible for positive treatmenteffects. This brings us to a second element of the proposed re-search agenda.

//. Identifying Research Therapy Practices ThatInfluence Outcome

This second approach involves the identification of factorsthat may account for the superior effects of research therapy,assuming, that the findings presented earlier hold up in futureresearch, and research therapy continues to show more positiveeffects than clinic therapy. Under this assumption, identifyingspecific lab-clinic differences that actually make a difference intreatment outcome may inform efforts toward lab-clinic col-laboration in at least two ways.4 First, some of the factors asso-ciated with beneficial effects of research therapy may point to-ward useful changes in clinic treatment procedures; assessingwhether such changes lead to improved outcomes of clinic treat-ment might then be made a part of the research agenda. Second,other factors that account for beneficial effects of research ther-apy may represent inherent differences between what is possiblein the lab and what is possible in real-life clinical practice; iden-tification of such factors may help researchers and clinicianslearn what changes are needed in lab-based interventions tomake them responsive to clinical realities.

To illustrate how investigators may use data from researchtherapy to probe for factors of both types, we offer here a series

of tests using data from a recent meta-analysis of child and ad-olescent outcome studies (Weisz, Weiss, et al., 1995). The testsfocus on eight hypotheses, eight possible reasons why researchtherapy may be more effective than clinic therapy. The eighthypotheses were derived primarily from salient differences be-tween research therapy (and its data base) and clinic therapy(and its data base). To test the hypotheses, we carried out aseries of analyses using a relevant subset of studies from theWeisz, Weiss, et al. (1995) meta-analysis. Of the 150 outcomestudies included in the meta-analysis, we excluded 47 that in-volved either mediated treatment (e.g., treatment limited toparent or teacher training) or treatment in which patients werea captive audience (e.g., inpatients in psychiatric hospitals orinmates in detention or correctional facilities). This left a totalof 103 studies, involving 163 treatment-control comparisons.5

The 103 studies encompassed a range of interventions(behavioral, including cognitive-behavioral; client centered; in-sight oriented), and of treated problems (internalizing, exter-nalizing, other).

As in most meta-analytic data sets, there was confoundingamong variables, and this made a clear test of any single hy-pothesis difficult to achieve. We addressed this problem to somedegree by complementing the individual tests with a simulta-neous elimination analysis designed to control for confoundingamong factors found to be significantly related to ES (discussedlater). However, given this and other limitations of the tests pre-sented here, it may be best to view them as having three primarykinds of value: (a) They identify eight potential hypotheses as towhy research therapy may be more effective than clinic therapy,hypotheses that warrant examination in future research, (b)they illustrate ways that tests of such hypotheses may be struc-tured and combined, and (c) they provide an initial wave ofevidence bearing on the validity of the hypotheses.

One other point should be made concerning the tests. Mostanalyses of group differences involving meta-analytic data setshave used the unweighted least squares (ULS) approach ini-tially used by Smith et al. (1980) in their landmark psychother-apy meta-analysis. Strictly speaking, this approach is only validif several assumptions are met, one of which is homogeneityof variance. Given the diversity of the studies in psychotherapymeta-analyses and the broad range of sample sizes involved, it

4 One possible contributor to the lab-vs.-Iab-clinic difference in treat-ment effects is a difference in methodological quality between lab andclinic studies to date. In a meta-analysis devoted entirely to method-ological factors, Weiss and Weisz (1990) found that the methodologicalrigor of child outcome studies was positively related to effect size. If laband clinic studies do differ in quality, that difference might well contrib-ute to differences in mean effects. For the most precise comparisons oflab and clinic treatment effects, lab and clinic studies of comparablemethodological rigor are needed, hence the need for increased attentionto careful outcome assessment in clinical practice settings.

5 A preliminary report of some of these analyses, based on a smallersample of 64 studies, appears in Weisz, Donenberg, Han, andKauneckis (1995). By the time the present article was finalized, a largerpool of outcome studies had been identified for the meta-analysis(Weisz, Weiss, et al., 1995). This larger pool afforded more reliable testsof the hypotheses than those reported previously in Weisz, Donenberg,et al. (1995). In addition, only the present analyses involve the use of aWLS approach, described in the text.


is quite unlikely that the homogeneity assumption is satisfied inmost meta-analytic data sets (for discussion, see Hedges & Ol-kin, 1985; Weisz, Weiss, et al., 1995). To address this problemin the present analyses, we tested group differences first withULS, and then with an approach generally viewed as more validfor data of this type (i.e., a weighted least squares [WLS] gen-eral linear models approach described by Hedges & Olkin[1985; see also Hedges, 1994]), with individual effect sizesweighted by the inverse of their variance.6 For efficiency of pre-sentation, let us simply note here that the ULS analyses yieldedno significant support for any of the eight hypotheses (the clos-est near miss was a marginal trend, p <. 10, toward larger effectsfor behavioral than nonbehavioral treatments). In what follows,we report the results of hypothesis tests using the WLS method,for which the significance of group differences is assessedthrough chi-square tests. Throughout the calculations, we con-sistently collapsed across outcome measures and treatmentgroups, up to the level of analysis. For example, we collapsedacross treatment groups except when we tested the effects ofbehavioral versus nonbehavioral therapies and different treat-ment groups within a study differed on that variable.

Possibilities not supported by the tests. Of the eight hypoth-esis, five were not supported in our WLS tests. Here we describethese hypotheses and how the tests were structured.7

1. Recent studies—better methods: Most of the clinic studiesavailable in the literature were published years ago, when ther-apy methods and assessment technology were relatively primi-tive: the research therapy studies have better outcomes becausethey are more recent and thus reflect better methodology. To ad-dress this hypothesis, we computed the correlation betweenyear of publication and ES in our sample of 103 studies. Thenegligible correlation of .05 (p > .40) was similar to that foundin previous meta-analyses in the child-adolescent area (—.05in Casey & Herman, 1985; .08 in Weisz et al., 1987). Thus,Hypothesis 1 was not supported by available correlational data.However, Weisz, Weiss, et al. (1995), in their recently com-pleted meta-analysis of 150 outcome studies, found that, al-though there was no overall main effect of publication year, theyear did interact significantly with child age and with child gen-der. There was some evidence that more recent studies showedlarger ES than older studies with samples that included primar-ily adolescents and in samples that involved female majorities.Therefore, although overall tests of Hypothesis 1 were not sup-portive here, the hypothesis may be valid for some subgroups oftreated youths. Further investigation certainly seems war-ranted. Finally, publication conventions change over time, andthere may be a trend toward more publication, in recent years,of treatment studies showing minimal effects. Such a trendwould work against detection of true improvements in therapyeffects over the years, at least in analyses that rely on publishedstudies as the data base.

2. Relative effectiveness of researcher-therapists and profes-sional clinicians: Clinicians are less effective than research ther-apists. Therapists in lab studies are often the investigators, grad-uate students, or research assistants, trained in the specifictreatment methods that will be used in their particular study.Are these "researcher-therapists" more effective than profes-sional clinicians? To address this question, we coded the 103studies for whether the therapists were clinicians (e.g., clinic

employees, psychiatric social workers, marriage and familycounselors, school counselors), researchers (e.g., university fac-ulty, study authors, graduate students, trained undergraduates),both types (i.e., one or more therapists from both categories),or not reported. Two raters, independently coding ten studiesinvolving 22 treatment-control comparisons, obtained a kappaof 0.86 for these judgments. When we compared the mean ESfor groups treated by clinicians with the mean ES for groupstreated by researchers, the difference was negligible, p > .50.

3. Pretherapy preparation (prep): Research therapy is moreeffective because the therapists have special training, just before theintervention, in the methods that will be used. As a third possibility,we tested the notion that research therapy is more effective becauseresearch therapists, more often than clinic therapists, are preparedfor their work through pretreatment training in the method of in-tervention that will be used. We classified the studies into threecategories: (a) Prep (explicitly stated or implied that such pre-treatment training was provided), (b) no prep (explicitly stated orimplied that there was no such training), or (c) no informationgiven. On these judgments, the two raters achieved a kappa of 1.00.When we compared mean ES for studies in Category a with themean ES for Category b, we found no evidence that outcomesdiffered as a function of whether therapists had pretreatment train-ing, p>.80.

4. Breadth of problem focus: Research therapy is more effectivebecause it tends to involve a focus on one type of child or one typeof child problem, whereas clinic therapy tends to be more diffuse,focusing on multiple child problems. It is part of the mandate inmany clinics that every youngster who has legitimate mentalhealth needs must be seen if clinic resources permit. Thus, unlikeresearch programs, clinics typically cannot limit their focus totreatment of one class of problems, for example, depression, or theanxiety disorders. Perhaps the freedom of research programs tofocus on single problems, or homogeneous groups, enhances theirmeasured effectiveness. To investigate this possibility, we used thetarget problem coding conducted for the Weisz, Weiss, et al.(1995) meta-analysis, from which the present 103 studies weredrawn. Tier 2 of that coding system classifies treated problems ata moderate level of specificity (e.g., depression, aggression) andfour coders for the meta-analysis achieved a mean kappa of 0.79in making these judgements. For the test of Hypothesis 4, we com-pared mean ES from studies involving a clear problem focus (i.e.,codes of delinquency, noncompliance, self-control [hyperactivity-impulsivity], aggression, phobias-anxiety, social withdrawal-iso-

6 To facilitate comparisons involving previous meta-analyses, most ofwhich used only unweighted means, we consistently report unweightedmean ES values in Figures 2, 3, and 4.

7 In preliminary analyses reported by Weisz, Donenberg, et al.(1995), 10 hypotheses were examined. Two of the 10 were excludedfrom the present series of analyses. In the Weisz, Donenberg, et al. arti-cle, unlike the present article, we considered whether effect size was re-lated to (a) the degree to which a specific, focused therapy approachwas used, as distinguished from multiple or diffuse methods, and to (b)duration of therapy: For the present analyses, we dropped the formerbecause the nature of our coding system for specificity made it difficultto interpret findings unambiguously; we dropped the latter because op-timum length of treatment seems to depend heavily on the type andseverity of treated problem, and too little information was available topermit proper control of such variables.


lation, depression, somatic problems [headaches], or cognitiveskills deficits) to mean ES for studies having a vague or diffuseproblem focus (i.e., multiple externalizing problems, multiple in-ternalizing problems, or multiple other problems). The twogroups showed a negligible difference (p > .80) in mean ES.

5. Structure: Research therapy is more effective because it ismore highly structured than clinic therapy. Compared withclinic therapy, research therapy seems more highly structured,with prearranged procedures used in a particular order, fre-quently guided by a treatment manual and quite often moni-tored for fidelity. Such structure may reduce flexibility, but itcould also enhance effectiveness by maximizing consistency inthe implementation of the planned intervention. To explore thispossibility, we classified studies into the following categories: (a)structured (i.e., treatment manual, monitoring of therapist con-sistency, treatment outline or detailed description, or audiovi-sual materials used to administer therapy), (b) unstructured(none of the aforementioned; e.g., "discussion groups"), and(c) not stated (inter-rater kappa for the two judges, 1.00). Whenwe compared structured therapy studies with unstructuredstudies, we found no significant difference in mean ES, p= .12.

Possibilities supported by the tests. Three of the eight pos-sibilities we examined were supported.

6. Relative treatability of research versus clinic cases: Cliniccases are more seriously disturbed, come from more dysfunc-tional families, or for some other reason, are more difficult totreat successfully than youngsters recruited for research therapy.Because so many of the youngsters treated in lab studies areanalog cases, recruited through screening in schools or throughadvertisements, and not likely to have ever been referred to aclinic, it seems likely that research therapy cases are less severelydisturbed than clinic therapy cases, on average. If treatmenteffects are more positive with less disturbed youngsters, thenthis difference in clientele may account for some of the lab-clinic difference in overall mean ES. To address the issue, weagain relied on coding done as a part of the recent meta-analysis(Weisz, Weiss, et al., 1995) from which our present 103 studieswere drawn. Our meta-analysis raters achieved a mean kappaof 0.89 for their judgments as to whether treated samples inthe various studies were clinical (i.e., the children were clinicreferred or would have received treatment regardless of thestudy) or analog (i.e., the children were recruited for the studyand would not have received treatment had it not been for thestudy) versus not stated. Our comparison revealed that analogsamples showed larger ES means than clinical samples (ESmeans, 0.50 vs. 0.29), x2 (1, N= 100) = 6.20,p < .05.

7. Conditions in clinic settings versus lab settings: Lab set-tings are generally more conducive to therapeutic gain than areclinic settings. Community clinics may not have the facilities orresources to match those of many lab researchers, who work tomake their setting and conditions so appealing that children andparents will be motivated to attend sessions faithfully. It is pos-sible that differences between the two settings have the effectof enhancing therapy effects in the lab or undermining therapyeffects in clinics. To explore this possibility, we coded studiesinto the following categories: (a) research setting (e.g., univer-sity lab, primary or secondary school, (b) clinic setting (e.g.,outpatient clinic), and (c) not stated [kappa for the two raters,0.78]. Comparison of clinic and research settings revealed a

higher mean ES for the latter (ES means, 0.50 vs. 0.16), x2 (1,N = 79) = 6.88, p < 0.01, and thus support for the hypothesis.

8. Behavioral versus nonbehavioral methods: Research ther-apy studies have better outcomes because they are more likelythan clinic studies to involve behavioral treatment methods.Kazdin, Bass, et al. (1990), in their analysis of characteristicsof therapy research, reported that the majority of treatmentoutcome research involves behavior modification (50% of allstudies surveyed) and cognitive-behavioral interventions(22%), with relatively little attention given to psychodynamic(1 %), family (4%), or eclectic (4%) interventions. By contrast,Kazdin, Siegel, and Bass's (1990) survey of child clinicians in-dicated that a majority viewed psychodynamic (59% ofrespondents), family (57%), and eclectic (73%) interventionsas useful most or all of the time. Consistent with this pattern,behavioral interventions were not well represented in the afore-mentioned nine clinic studies reviewed, but were very well rep-resented in each of the four broad-based meta-analyses de-scribed earlier. If behavioral interventions were more effectivewith young people than nonbehavioral, such an effect mighthelp account for the apparent superiority of research therapyover clinic therapy.

To address this issue, we relied on coding done for the meta-analysis by Weisz, Weiss, et al. (1995) that was the source of the103 studies used here. The meta-analysis raters showed a meankappa of 0.83 in classifying treatments into the categories be-havioral, nonbehavioral, and other. In support of the hypothe-sis, behavioral treatments produced larger mean ES than non-behavioral (ES means, 0.50 vs. 0.25), x2 (1, N = 103) = 7.64,p < .01. This finding is consistent with reports in the three meta-analyses in which behavioral and nonbehavioral treatmentshave been compared (i.e., Casey & Berman, 1985; Weisz et al.,1987, Weisz, Weiss, et al., in press). In all three, the initial anal-yses showed higher mean ES for behavioral than nonbehavioralinterventions (but see Casey & Berman, 1985, for further tests,and see Weiss & Weisz, 1995, regarding proper interpretationof such findings). If behavioral methods truly generate morepositive effects than nonbehavioral methods, the fact that be-havioral methods are so commonly used in research therapy,and so infrequently used in clinic therapy, may help explain thesuperior outcomes of research therapy.

Assessing the Independent Contribution of the ThreeSignificant Factors

The three factors that showed significant relations with ESwere correlated with one another. Consequently, we sought toassess the effect of each factor independently of the other two.This was accomplished by means of a standard regressionmodel with the three factors as independent variables and ESas the dependent variable. With this simultaneous eliminationprocedure, the effect of lab versus clinic setting was reduced tononsignificance, (adjusted ES means, 0.30 vs. 0.27), x2 (1, N =78) = 0.35, p > .55. The relation between ES and the analogversus clinical case variable remained significant (adjustedmeans, 0.44 vs. 0.13), x

2 (1, N = 78) = 4.68, p < .05; andthe relation between ES and behavioral versus nonbehavioraltreatments also remained significant (adjusted means, 0.42 vs.0.14), x

2 (1, N = 78) = 7.22, p < .01. What these findings


suggest is that the apparent effect of clinical setting was an arti-fact of that variable's confounding with the analog-clinical casevariable or the behavioral-nonbehavioral variable, but that theeffects of these latter two variables on ES did not result fromsuch confounding.

Using Findings to Inform Lab-Clinic Collaboration

As noted earlier, identifying lab-clinic differences that are sig-nificantly related to treatment effects may inform lab-clinic co-operation in at least two ways. The findings of these present testsillustrate both. First, some factors associated with positiveeffects of research therapy may help point toward beneficialchanges in clinic treatment procedures. The finding that behav-ioral treatments yield larger effects than nonbehavioral suggestsa change in clinic practice that might yield real benefits. Otherfactors linked to beneficial effects of research therapy may re-flect relatively unchangeable aspects of clinic-based treatmentand may, thus, point toward changes needed in lab-based thera-pies to help them accommodate to clinic realities. Our findingthat analog cases showed bigger treatment gains than clinic-re-ferred youths suggests that the success of lab studies may restpartly on the relatively low levels of disturbance in their re-cruited samples. In fact, after youth recruitment, laboratory re-searchers frequently apply additional exclusionary criteria totheir pool of recruits (e.g., disallowing major comorbiddiagnoses), in an effort to select the very best candidates fortheir particular treatment, and this raises further questionsabout the generalizability of the effects obtained. If lab researchfindings are to be optimally useful to practitioners, the lab treat-ments may need to be modified to make them applicable to abroader range of youths than they currently accommodate. Tosummarize, optimum lab-clinic collaboration is apt to involvechanges not only in clinic procedures but also in the ways labo-ratory interventions are conducted.

Comparing the Clinic Studies to Inpatient andResidential Add-On Studies

Although we did find that mean ES was higher for analog thanclinic samples, there were small-to-medium-range (J. Cohen,1992) treatment effects with clinic samples (see also Weisz etal., 1987, Weisz, Weiss, et al., 1995). Another indication thatstructured interventions can be effective with clinic-referredgroups in clinical settings comes from an examination of 10studies that came to light in our search for clinic studies(discussed earlier) but that did not quite fit the clinic study cat-egory. In these 10 studies, which fall into what might be calledan "add-on" genre, specific treatment interventions were addedto the standard program of various inpatient or residential fa-cilities (excluding juvenile correctional centers). Treatmentgroups were compared with control groups of youths in thesame facility who did not receive the special treatmentprogram.

We calculated an ES (or estimate, following Smith et al.[ 1980 ] and Glass et al. [ 1981 ]) for each treatment-control com-parison (i.e., the focused intervention vs. the standard institu-tional program) in each of the studies. Across the 10 studies(with one ES per study), the mean ES was 0.76; after treatment,

the average youngster receiving one of the special interventionswas at the 78th percentile of the control group, averaging acrossmeasures of adjustment. As Figure 4 shows, this mean ES valuewas very similar to the ES mean of the four broad-based childmeta-analyses reported earlier; and, of course, this add-onmean ES was markedly higher than the clinic study mean ES, p<.01.

In interpreting the mean effect shown in Figure 4, it is impor-tant to note that inpatient and residential settings offer certainadvantages over outpatient settings as test sites; for example,inpatient youngsters are a captive audience for the treatment,and the controlled settings help to minimize the influence ofsuch treatment-undermining influences as family disorganiza-tion and deviant peer groups. Nonetheless, the difference be-tween the mean ES for clinic studies and add-on studies maybe instructive, given other dimensions along which the add-onstudies resemble and differ from the clinic studies. Importantareas of resemblance are that both types of studies (a) involvesamples not of recruited children but of youngsters disturbedenough that they had been referred for treatment (in fact, someof these inpatient-residential samples seem likely to have beenmore disturbed than the [ outpatient ] clinic study samples), (b)took place in clinical settings, and (c) primarily involved ther-apy conducted by practicing clinicians. However, most of theadd-on studies, unlike most clinic studies, involved behavioralinterventions. Taken together, the studies illustrate that inter-vention may be made effective with significantly disturbed,clinic-referred youngsters, treated in clinical settings. The ques-tion of whether this may be done successfully in nonresidentialsettings, however, brings us to a third proposal for research link-ing lab and clinic.

///. Exporting Treatments From Labs to ClinicalPractice Sites

A third potentially valuable research strategy involves ex-porting well-developed research therapy programs from the lab-oratories where they have been created and tested to the provingground of service-oriented clinics. This seems likely to set inmotion a productive cycle of outcome assessment, refinementsin the procedures to fit clinic conditions, further outcome as-sessment, and so forth. A number of the studies shown in Figure4 involved efforts to test structured treatment programs in in-patient settings and with considerable success. Other articles inthis special section illustrate that the exporting of lab proce-dures to outpatient clinic and community settings is also un-derway, albeit in different ways for different treatmentprograms.

The research literature includes a number of useful exam-ples. Some studies in the literature involve implementation ofresearch therapy programs with clinic-referred youths in clinicsettings. Examples include Fehlings, Roberts, Humphries, andDawe's (1991) study of cognitive-behavioral treatment of out-patient children with attention deficit-hyperactivity disorder(ADHD); Seymour, Brock, During, and Poole's (1989) studyof operant intervention for sleep disruptions in outpatient chil-dren; and McKenna and Gaffney's (1982) study of social skillstraining for a heterogeneous group of outpatient youths.

Other studies involve implementation of research therapy


FINCH JENNINGSETAL. & DAVIS(1975) (1977)

KAZDINETAL.(1987)

KENDALLETAL.(1978/9)

KOLKOETAL(1990)

KROPETAL.(1990)

MORGAN NAHME-ETAL HUANG(1991) ETAL.

(1977)

REARDON SNYDER&TOSI 4 WHITE(1977) (1979)

Figure 4. Estimated effect sizes for 10 "add-on" studies of child and adolescent psychotherapy in hospitaland residential settings.

programs with clinic-referred youngsters, or youths who clearlywarrant referral, but with the treatment provided in university-based clinics or otherwise outside conventional service-orientedclinics; examples include Kendall's (1994) cognitive-behav-ioral treatment program for children with anxiety disorders,Lovaas and his colleagues' operant conditioning program forautistic children (Lovaas, 1987; McEachin, Smith, & Lovaas,1993), and Lewinsohn, Clarke, Hops, and Andrews' (1990)cognitive-behavioral intervention for depressed adolescents(see also Clarke et al., 1992).

A slightly different but highly relevant genre of research isthe effort by several investigators (e.g., Lee & Haynes, 1976;Stewart, Vockell, & Ray, 1986) to bring structured interven-tions directly to juvenile offenders in the community (i.e., notnecessarily through clinics). Strong positive effects of such aneffort, focused on the family and social system, have been dem-onstrated by Henggeler and his colleagues (Henggeler, Melton,& Smith, 1992).

Thus, several teams of investigators have begun to show thatbeneficial treatment effects can be sustained when well-devel-oped interventions are taken out of the security of the lab andtested with seriously disturbed youths in clinic and communitysettings. Ultimately, the success of these efforts may be the most

persuasive argument for the value of research to clinicalpractitioners.

Obstacles to Lab-Clinic Collaboration

Thus far, we have discussed three broad strategies for linkingresearch and clinical practice. Each of the strategies may be use-ful, but none is apt to be simple. Each strategy may requirereconsideration—and, ultimately, restructuring—of how chil-dren are treated in many clinics; and it would be naive to thinkthat such changes would be painless. Several particularly formi-dable obstacles can be anticipated, some philosophical-attitu-dinal and some quite practical. We offer a few examples of each.

1. Belief that frank scrutiny of therapy outcomes is harmfulto the discipline. Perhaps the most sweeping attitudinal obstacleto the kinds of change we are considering here is the belief thatfrank discussion of the evidence on treatment outcomes mayharm our discipline. Some believe, for example, that presentingfindings that are not uniformly favorable may undermine ourimage in the public eye. We suggest, by contrast, that publicconfidence may ultimately be strengthened by a recognitionthat this are a self-monitoring, self-correcting discipline, thatfindings (both positive and negative) are reported openly, and


that science is used to improve practice. By its very nature, sci-entific evidence on outcomes cannot all be good news. However,it must not be forgotten that much of the news from the currentinformation base is positive: Meta-analyses of over 300 outcomestudies indicate that the most carefully tested interventions forchildren and adolescents have generally beneficial effects; more-over, a number of efforts to implement lab-tested interventionswith disturbed children in clinic and community settings haveproduced positive outcomes. On the other hand, some evidencedoes raise a concern that the rather strong positive effects shownwith lab-tested interventions may not be replicated in everydayclinic practice. If the concern proves to be well founded, surelyall who share an interest in the well being of children and fami-lies would want it to be addressed.

2. Belief that psychotherapy is art, not science. Another po-tential obstacle to the kinds of lab-clinic collaboration pro-posed here is the view that psychotherapy is an art and that thepractice of this art should not be shaped according to researchfindings. Of course, some researchers may reply that this view isprecisely the problem with psychotherapy (i.e., that its practi-tioners have failed to incorporate scientific evidence, and totheir own detriment). The tension between these two view-points defines much of the gap between clinic and lab that is thefocus of this article. In principle, it should be possible to iden-tify a middle ground between the two perspectives (e.g., to ap-preciate elements of art in psychotherapy without declaringtreatment immune to tests of efficacy or concluding that scien-tific findings are irrelevant). Ultimately, for the consuming pub-lic, what may matter most is a basic empirical question: Arethere reliable differences in outcome between treatments thathave been developed and tested through research and those thathave not?

3. Belief that current clinical practice works well, but in waysthat outcome research cannot measure. Some therapists sin-cerely believe that their interventions are effective but in waysnot measurable by means of current research methods. Thisbelief can take a number of forms, but one example is the viewthat effects of some forms of psychotherapy are so complex,subtle, and private as to defy objective assessment. Therapiesaimed, for instance, at promoting insight, self-awareness, or en-hanced identity, might be subject to this claim. The argumentcan be made unaddressable if proponents label as inadequateany outcome measure researchers may propose. On the otherhand, much could be accomplished if proponents and research-ers could work together to identify mutually acceptable mea-sures of treatment outcome. Such a partnership may be increas-ingly critical. With changes in the health care system, third-party payers and the public alike are apt to require evidence ofthe benefits for which they are asked to pay. Everyone will beserved best if that evidence takes a form that is judged fair andappropriate by both researchers and clinicians.

4. Belief that outcome research, with its narrow focus andmanualized treatments, is not relevant to the complex cases seenin clinics. Many research therapy outcome studies have im-posed a rather pristine simplicity and linearity on the treatmentprocess (e.g., by focusing treatment on one or two focal prob-lems such as aggression and by detailing steps of treatment in amanual). It does seem likely that applying lab-tested methodsto complex clinic cases will reveal limitations and necessitate

adjustments in those methods; researchers may learn a greatdeal from this process about how to make their lab-developedinterventions more responsive to the conditions of seriously dis-turbed children in clinics. On the other hand, although it is truethat real clinic cases can be quite complex, it must also be notedthat a number of successful research therapy studies involvedtreatment of real clinic cases. The evidence reviewed earlier sug-gests that the use of focused behavioral techniques is associatedwith particularly positive treatment effects. In some cases, aim-ing for specific behavioral changes may be more effective thantrying to encompass all the complexity of a very troubled childat once.

5. Devotion to conceptually appealing, personal theoreticalorientations. For many years, in many of our training programs,clinical training has encouraged trainees to develop a specific,coherent theoretical orientation. Articulating this orientationoften becomes part of written qualifying exams, internship andjob interviews, and oral examinations for state licensure. As cli-nicians, the assessments, treatments, and supervision of train-ees are all expected to reflect our theoretical orientation; inthese and other ways, that theoretical orientation may becomeso central to the professional and personal identity of cliniciansthat the idea of giving it up borders on the unthinkable. Unfor-tunately, it seems inevitable that the theoretical orientationsheld by some clinicians will not fare well under empirical scru-tiny, and the therapies derived from these orientations will notbe shown to be effective. The therapists for whom this is truewill face a difficult choice: Persist in the preferred orientationand its methods despite a lack of supporting evidence, or learnnew ways of construing and intervening with clinical problems.The latter choice will not be easy; in some cases it will meanrefashioning or even replacing hard-won, familiar, and concep-tually appealing ways of thinking and intervening. Those clini-cians who are willing to take this difficult step in response to theevidence will deserve the respect of their peers and the gratitudeof their clients.

6. Reluctance of third-party payers to fund restructuringefforts. We turn now to more practical concerns, beginning withone of the most practical of all. Assuming that therapists andclinic administrators were inclined to reorient clinic proce-dures in response to empirical evidence, who would pay for theprocess? Third-party payers seem more inclined to pay for ser-vices than for the kinds of time-consuming restructuring andtraining that could be required to link mental health care to itsscientific base. On the other hand, as noted earlier, cliniciansface the growing possibility that third-party payments will belinked to evidence of efficacy in the future. To the extent thatthis is true, clinics and clinicians whose work has been maderesponsive to the evidence of outcome research may find thattheir position as providers has been strengthened. Clinic admin-istrators may find that it makes good sense to promote neededchanges through such mechanisms as seminars, in-serviceworkshops, and continuing education credits.

7. Undersupply of clinicians from training programs who arewell-versed in the use of empirically supported methods. Ofcourse, seminars and workshops can only go so far to instill ex-pertise in empirically supported interventions. Such mecha-nisms are no substitute for an ample supply of clinicians trainedto carry out such interventions from the day they begin work.


Currently, the supply of such clinicians appears to be very lim-ited. Despite the discipline's commitment to the scientist-prac-titioner model, clinical training programs and accredited in-ternships have not, for the most part, limited their training to,or even focused their training on, those forms of therapy thathave scientific support; nor have trainees generally been taughtto use the empirical literature in selecting treatments. As a re-sult, relatively few graduates from clinical psychology programsare currently in a position to lead clinics into a world of empir-ically supported interventions. This state of affairs could bechanged but only through substantial modifications in the ori-entation and policies of most of the graduate training and in-ternship programs. Adjustments in these programs, however,may help to create a new generation of scientifically trained cli-nicians poised to guide clinics of the 21 st century in the use ofinterventions that have been shown to work.

8. Difficulty of finding and learning proven interventions forthe diverse array of problems seen in clinics. Even therapistsfully committed to using only empirically supported treatmentscould face a daunting challenge. Such therapists could be over-whelmed by the task of finding such treatments for the diversechildren who enter a typical community clinic or HMO pro-gram in a single month. Beyond mere identification of suchtreatments lie the tasks of obtaining the treatment manuals andgaining sufficient training and supervision to be truly proficientin and effective with the manualized method. It seems to us thatthis very real problem can only be addressed effectively throughchanges at the lab and clinic ends of the process. Researcherswho develop and test treatment programs need to find ways tomake these programs accessible to therapists outside the lab,and clinics may need to move away from the notion of the "gen-eralist clinician" and away from assigning cases on a "who's upnext?" basis. Indeed, it may never have been very realistic toassume that a well-trained clinician should be able to treat thefull range of youngsters who come through the clinic door. Inthe future, clinics may be increasingly organized into specialtysubclinics, each with its team of clinicians trained to do a fewthings particularly well and to train and supervise others in thesame specialized skills.

9. Lack of proven treatments for some problems. The clini-cian who seeks to use only empirically supported treatmentswill soon find that for some of the problems seen in clinics thereis no tested and proven treatment. This state of affairs wouldnot necessarily mean that children with these problems shouldbe denied treatment; but it may call for truth in advertising. Itmay be only fair, for example, for family members to knowwhether the treatment proposed for their child has been testedand shown to be effective (and with what groups, under whatconditions) or is an experimental approach designed for theirchild in the absence of an empirically based treatment. In thelatter case, when families agree to proceed, the experience cancontribute data on how children change when exposed to theexperimental treatment, and these data may contribute to anongoing lab-clinic collaboration in treatment development.

Summary and Conclusion: Bidirectional Benefits ofLab-Clinic Collaboration

To summarize, we reviewed evidence suggesting that the ben-eficial effects of child psychotherapy seen in experimental stud-

ies may not be well replicated in clinic practice. To address thissituation, we proposed three kinds of research at the lab-clinicinterface, designed to promote enhanced treatment effects withreferred children in clinical practice. Then, we outlined someproblems likely to be confronted in efforts to link research find-ings and clinic practice, offering a perspective on each of theproblems. Overall, we have stressed the importance of franklyexamining all available evidence, both favorable and unfavor-able, in efforts to strengthen the impact of child interventions.The discipline grows as researchers and clinicians learn whereimprovement is needed and inculcate new skills in response tonew evidence.

One other point should be stressed: The benefits of lab-cliniccollaboration seem likely to flow in both directions. The evi-dence reviewed here suggests that the application of researchtherapy procedures and findings may help to enhance the effectsof clinical practice with referred children. Conversely, attemptsto bring some of the more pristine laboratory procedures intoclinics may prompt significant adjustments in those proce-dures; in the process, changes needed to make lab-tested thera-pies effective in clinical practice may be identified. Clinical re-searchers may have a great deal to learn from practicing clini-cians, just as clinicians may learn useful new approaches fromthe research community. If the obstacles to researcher-cliniciancollaboration can be overcome, both groups may profit, and tothe ultimate benefit of the children and families who seek help.

References

Ashcraft, C. W. (1971). The later school adjustment of treated and un-treated emotionally handicapped children. Journal of School Psy-chology, 9, 338-342.

Baer, R. A., & Nietzel, M. T. (1991). Cognitive and behavioral treat-ment of impulsivity in children: A meta-analytic review of the out-come literature. Journal of Clinical Child Psychology, 20, 400-412.

Casey, R. J., & Berman, J. S. (1985). The outcome of psychotherapywith children. Psychological Bulletin, 98, 388-400.

Catron, T, & Weiss, B. (1994). The Vanderbilt school-based counselingprogram: An interagency, primary-care model of mental health ser-vices. Journal of Emotional and Behavioral Disorders, 2, 247-253.

Clarke, G., Hops, H., Lewinsohn, P. M., Andrews, J., Seeley, J. R., &Williams, J. (1992). Cognitive-behavioral group treatment of ado-lescent depression: Prediction of outcome. Behavior Therapy, 23,341-354.

Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159.

Cohen, L. (1979). The research readership and information source re-liance of clinical psychologists. Professional Psychology, 10, 780-785.

Cohen, L., Sargent, M., & Sechrest, L. (1986). Use of psychotherapyresearch by professional psychologists. American Psychologist, 41,198-206.

De Fries, Z., Jenkins, S., & Williams, E. C. (1964). Treatment of dis-turbed children in foster care. American Journal of Orthopsychiatry,54,615-624.

Durlak, J. A., Fuhrman, T, & Lampman, C. (1991). Effectiveness ofcognitive-behavior therapy for maladapting children: A meta-analy-sis. Psychological Bulletin, 110, 204-214.

Dush, D. M., Hirt, M. L., & Schroeder, H. E. (1989). Self-statementmodification in the treatment of child behavior disorders: A meta-analysis. Psychological Bulletin, 106, 97-106.

Elliot, R. (1983). Fitting process research to the practicing psychother-apist. Psychotherapy: Theory, Research, and Practice, 20, 47-55.


Fehlings, D. L., Roberts, W., Humphries, T., & Dawe, G. (1991). At-tention deficit hyperactivity disorder: Does cognitive behavior ther-apy improve home behavior? Developmental and Behavioral Pediat-rics, 12, 223-228.

Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in so-cial research. Beverly Hills, CA: Sage.

Hazelrigg, M. D., Cooper, H. M., & Borduin, C. M. (1987). Evaluatingthe effectiveness of family therapies: An integrative review and analy-sis. Psychological Bulletin, 101, 428-442.

Hedges, L. V. (1994). Fixed effects models. In H. Cooper & L. B.Hedges (Eds.), The handbook of research synthesis (pp. 285-300).New York: Sage.

Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis.Orlando, FL: Academic Press.

Henggeler, S. W., Melton, G. B., & Smith, L. A. (1992). Family preser-vation using multisystemic therapy: An effective alternative to incar-cerating serious juvenile offenders. Journal of Consulting and ClinicalPsychology, 60, 953-961.

Jacob, T., Magnussen, M. G., & Kemler, W. M. (1972). A follow-up oftreatment terminators and remainers with short-term and long-termsymptom duration. Psychotherapy: Theory, Research, and Practice,9, 139-142.

Jennings, R. L., & Davis, C. S. (1977). Attraction-enhancing client be-haviors: A structured learning approach for "non-YAVIS, Jr." Journalof Consulting and Clinical Psychology, 45, 135-144.

Kazdin, A. E. (1992). Research design in clinical psychology (2nd. ed.).New York: Macmillan.

Kazdin, A. E. (1994). Behavior modification in applied settings (5thed.). Pacific Grove, CA: Brooks/Cole.

Kazdin, A. E., Bass, D., Ayers, W. A., & Rodgers, A. (1990). Empiricaland clinical focus of child and adolescent psychotherapy research.Journal of Consulting and Clinical Psychology, 58, 729-740.

Kazdin, A. E., Esveldt-Dawson, K., French, N. H., & Unis, A. S.(1987). Problem-solving skills training and relationship therapy inthe treatment of antisocial child behavior. Journal of Consulting andClinical Psychology, 55, 76-85.

Kazdin, A. E., Siegel, T. C., & Bass, D. (1990). Drawing on clinicalpractice to inform research on child and adolescent psychotherapy:Survey of practitioners. Professional Psychology: Research and Prac-tice, 21. 189-198.

Kendall, P. C. (1994). Treating anxiety disorders in children: Results ofa randomized clinical trial. Journal of Consulting and Clinical Psy-chology, 62, 100-110.

Kendall, P. C., & Finch, A. J., Jr. (1978). A cognitive-behavioral treat-ment for impulsivity: A group comparison study. Journal of Consult-ing and Clinical Psychology, 46, 110-118.

Kendall, P. C., & Finch, A. J., Jr. (1979). Analyses of changes in verbalbehavior following a cognitive-behavioral treatment for impulsivity.Journal of Abnormal Child Psychology, 7, 455-463.

Kolko, D. J., Loar, L. L., & Sturnick, D. (1990). Inpatient social-cog-nitive skills training groups with conduct disordered and attentiondeficit disordered children. Journal of Child Psychology and Psychi-atry, 31, 737-748.

Krop, H., Calhoon, B., & Verrier, R. (1971). Modification of the "self-concept" of emotionally disturbed children by covert reinforcement.Behavior Therapy, 2, 201-204.

Kupfersmid, J. (1988). Improving what is published: A model in searchof an editor. American Psychologist, 43, 635-642.

Lee, R., & Haynes, N. M. (1976). Counseling juvenile offenders: Anexperimental evaluation of Project CREST. Community MentalHealth Journal, 14, 267-271.

Lehrman, L. J., Sirluck, H., Black, B. J., & Click, S. J. (1949). Successand failure of treatment of children in the child guidance clinics of the

Jewish Board of Guardians, New York City. Jewish Board of Guard-ians Research Monographs, No. 1.

Levitt, E. E., Beiser, H. R., & Robertson, R. E. (1959). A follow-upevaluation of cases treated at a community child guidance clinic.American Journal ofOrthopsychiatry, 29, 337-347.

Lewinsohn, P. M., Clarke, G. N., Hops, H., & Andrews, J. (1990). Cog-nitive-behavioral treatment for depressed adolescents. Behavior Ther-apy, 27,385-401.

Lovaas, O. I. (1987). Behavioral treatment and normal educational andintellectual functioning in young autistic children. Journal of Con-sulting and Clinical Psychology, 55, 3-9.

Luborsky, L. (1972). Research cannot yet influence clinical practice.In A. Bergin & H. Strupp (Eds.), Changing frontiers in the science ofpsychotherapy (pp. 120-127). Chicago: Aldine.

Mann, C. (1990). Meta-analysis in the breech. Science, 249, 476-480.McEachin, J. J., Smith, T, & Lovaas, O. I. (1993). Outcome in adoles-

cence of autistic children receiving early intensive behavioral treat-ment. American Journal of Mental Retardation.

McKenna, J. G., & Gaffney, L. R. (1982). An evaluation of group ther-apy for adolescents using social skills training. Current PsychologicalResearch, 2, 151-160.

Moran, G., Fonagy, P., Kurtz, A., Bolton, A., & Brook, C. (1991). Acontrolled study of the psychoanalytic treatment of brittle diabetes.Journal of the American Academy of Child and Adolescent Psychiatry,30, 926-935.

Morrow-Bradley, C., & Elliott, R. (1986). Utilization of psychotherapyresearch by practicing psychotherapists. American Psychologist, 41,188-197.

Nahme-Huang, L., Singer, D. G., Singer, J. L., & Wheaton, A. B.(1977). Imaginative play training and perceptual-motor interventionwith emotionally disturbed hospitalized children. American JournalofOrthopsychiatry, 47, 238-249.

Nelson, W. M. Ill, & Montgomery, L. E. (1975). Modification of animpulsive cognitive tempo in emotionally disturbed boys. Journal ofAbnormal Child Psychology, 3, 49-52.

Orlinsky, D., & Howard, K. (1978). The relation of process to outcomein psychotherapy. In S. Garfield & A. Bergin (Eds.), Handbook ofpsychotherapy and behavior change: An empirical analysis (2nd ed.,pp. 283-330). New York: Wiley.

Parloff, M. B. (1980). Psychotherapy and research: An anaclitic depres-sion. Psychiatry, 43, 279-293. Paul, G. (1967). Outcome research inpsychotherapy. Journal of Consulting Psychology, 31, 109-118.

Prout, H. T, & DeMartino, R. A. (1986). A meta-analysis of school-based studies of psychotherapy. Journal of School Psychology, 24,285-292.

Raw, S. D. (1993, March). Does psychotherapy research teach us any-thing about psychotherapy? The Behavior Therapist, pp. 75-76.

Reardon, J. P., & Tosi, D. J. (1977). The effects of rational stage di-rected imagery on self-concept and reduction of psychological stressin adolescent delinquent females. Journal of Clinical Psychology, 33,1084-1092.

Russell, R. L., Greenwald, S., & Shirk, S. R. (1991). Language changein child psychotherapy: A meta-analytic review. Journal of Consult-ing and Clinical Psychology, 59, 916-919.

Seymour, F. W., Brock, P., During, M., & Poole, G. (1989). Reducingsleep disruptions in young children: Evaluation of therapist-guidedand written information approaches: A brief report. Journal of ChildPsychology and Psychiatry, 30, 913-918.

Shapiro, D. A., & Shapiro, D. (1982). Meta-analysis of comparativetherapy outcome studies: A replication and refinement. PsychologicalBulletin, 92, 581-604.

Shepherd, M., Oppenheim, A. N., & Mitchell, S. (1966). Childhoodbehaviour disorders and the child-guidance clinic: An epidemiologi-cal study. Journal of Child Psychology and Psychiatry, 7, 39-52.


Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapyoutcome studies. American Psychologist, 32, 752-760.

Smith, M. L., Glass, G. V., & Miller, T. L. (1980). Benefits of psycho-therapy. Baltimore: Johns Hopkins University Press.

Smyrnios, K. X., & Kirkby, R. J. (1993). Long-term comparison ofbrief versus unlimited treatments with children and their parents.Journal of Consulting and Clinical Psychology, 61, 1020-1027.

Snyder, J. J., & White, M. J. (1979). The use of cognitive self-instruc-tion in the treatment of behaviorally disturbed adolescents. BehaviorTherapy, 10, 227-235.

Stewart, M. J., Vockell, E., & Ray, R. E. (1986). Decreasing court ap-pearances of juvenile status offenders. Social Casework: The Journalof Contemporary Social Work, 67, 74-79.

Strupp, H. H. (1989). Psychotherapy: Can the practitioner learn fromthe researcher? American Psychologist, 44, 717-724.

Suinn, R. M. (1993, February). Practice—It's not what we preached.The Behavior Therapist, pp. 47-49.

Weiss, B., & Weisz, J. R. (1990). The impact of methodological factorson child psychotherapy outcome research: A meta-analysis for re-searchers. Journal of Abnormal Child Psychology, 18, 639-670.

Weiss, B., & Weisz, J. R. (1995). Relative effectiveness of behavioraland nonbehavioral child psychotherapy. Journal of Consulting andClinical Psychology, in press.

Weisz, J. R., Donenberg, G. R., Han, S. S., & Kauneckis, D. (1995).Child and adolescent psychotherapy outcomes in experiments and inclinics: Why the disparity? Journal of Abnormal Child Psychology,23, 83-106.

Weisz, J. R., & Weiss, B. (1989). Assessing the effects of clinic-basedpsychotherapy with children and adolescents. Journal of Consultingand Clinical Psychology, 57, 741-746.

Weisz, J. R., & Weiss, B. (1993). Effects of psychotherapy with childrenand adolescents. New "York: Sage.

Weisz, J. R., Weiss, B., Alicke, M. D., & Klotz, M. L. (1987). Effective-ness of psychotherapy with children and adolescents: A meta-analysisfor clinicians. Journal of Consulting and Clinical Psychology, 55,542-549.

Weisz, J. R., Weiss, B., & Donenberg, G. R. (1992). The lab versusthe clinic: Effects of child and adolescent psychotherapy. AmericanPsychologist, 47, 1578-1585.

Weisz, J. R., Weiss, B., Han, S., Granger, D. A., & Morton, T. (1995).Effects of psychotherapy with children and adolescents revisited: Ameta-analysis of treatment outcome studies. Psychological Bulletin,7/7,450-468.

Wilson, G. T. (1985). Limitations of meta-analysis in the evaluation ofthe effects of psychological therapy. Clinical Psychology Review, 5,35-47.

Witmer, H. L., & Keller, J. (1942). Outgrowing childhood problems: Astudy in the value of child guidance treatment. Smith College Studiesin Social Work, 13, 74-90.

Received January 6, 1994Revision received January 11, 1995

Accepted January 11, 1995 •

bridging the gap between laboratory and clinic in...

Documents