multiple indicators of success in residential youth care and treatment

Amer. J . Orthopsychiat. % ( I ) , January 1988

MULTIPLE INDICATORS OF SUCCESS

James K. Whittaker, Ph.D., Edward J. Overstreet, M.S.W., Anthony Grasso, M.S.W., Tony Tripodi, D.S. W., Francis Boylan, M.S.W.

Based on a large, ongoing empirical research effort to determine factors associated with the successful community adjustment of troubled adolescents leaving residential treatment, this paper focuses on multiple indicators of success measured at multiple points of time in the treatment process. Implications for research and clinical practice are discussed.

ince Matsushima’s classic article (1965), S clinicians, researchers, and program managers have struggled to define an adequate criterion for success in residential youth care and treatment. Sometimes these efforts have focused on attainment of treatment goals at the time of discharge and sometimes on adaptation in the postplacement environment. Recent reviews of these outcome studies underscore the importance of the postdischarge environment as a powerful intervening factor in successful adaptation and community integration (Whittaker & Pecora, 1984). Programs ap- pear to be reasonably successful in altering youth behavior during the course of residential treatment, but less successful in in- suring the maintenance of those gains and their generalization to the postdischarge en- vironments of family, school, and community (Nelson, Singer, & Johnson, 1978; Jones, Weinrott, & Howard, 1981). While such findings caused some to question the continuation of residential placement as a

treatment of choice, recent reviews of home-based treatment services and other permanency options, such as adoption, raise serious questions about their advantages as a substitute for residential provision (Barth & Berry, 1987). Renewed efforts are needed to address the question of what can reasonably be expected from residential youth care and treatment as one option in the service continuum: what range of success indicators? with what type of youth? at what level of service intensity? measured where?

Nearly a decade before Matsushima, Ger- shenson (1956) delineated the challenging methodological issues involved in specify- ing and evaluating as complex a service option as residential treatment. As subse- quent reviewers have noted, much of the current research on residential youth care and treatment continues to be plagued by these same problems (Durkin & Durkin, 1975; Whittaker, 1979; Whittaker & Pe- Cora, 1984). Typically, these include: l) failure to specify differential levels of risk

Submitted to the Journal in August 1987. Authors are at: University of Washington, Seattle (Whittaker); Boysville of Michigan, Southfield (Overstreet, Grasso. Boylan); University of Pittsburgh (Tripodi).

1 43 0 1988 American Orthopsychiatric Association, Inc.

144 RESIDENTIAL YOUTH CARE

(youth-family-community factors) at intake; 2) failure to obtain baselines on target behavior; 3) absence of control or contrast groups; 4) restricted measurement point or restricted indicators of success; 5 ) services inadequate in frequency, intensity, and duration to achieve desired postplacement outcomes; 6) inadequate in-program tracking and monitoring and community follow-up and 7) insufficient specification of treatment procedures and client or program goals. While the ongoing research program at Boysville of Michigan is designed, ultimately, to focus on these and related questions, our purpose in this brief paper is to address the criterion question itself. Specif- ically, we propose differential measures of success at four different times in the placement process: intake, during treatment, discharge, and postplacement. On the basis of our preliminary research to date and of a selected review of other evaluation efforts, we identify potential indicators of success and measures appropriate to routine data- gathering efforts.

RESEARCH PROGRAM

Boysville of Michigan is the state’s larg- est private residential agency for troubled adolescents. Presently, over 400 youth are served on a central campus and in community-based group homes throughout the state. Additional services include special- ized foster care and in-home treatment. A group work program, based on a modified version of Positive Peer Culture, is augmented by an active family work program, special education, and related services. Youth enter the program through a variety of service streams-juvenile justice, social services, mental health-and with a wide range of presenting problems and condi- tions. In 1982, Boysville introduced a com- puterized management information system, including routinely and repeatedly administered standardized measures, which con- tinuously tracks data on individual youth, their families, and community factors from

intake through community follow-up. A national research advisory committee guides the course of the total research effort. Rep- resentative studies completed or under way include: research utilization in youth care (Grasso, Epstein, & Tripodi, 1987); youth and family stress, coping, and adaptation (McCubbin, 1987) and intake patterns (Whit- taker, Fine, & Grasso, 1987).

INDICATORS OF SUCCESS

The routinely collected data provided by this information system suggested the potential for tracking multiple indicators of success at different points in the placement process (Tripodi, Fellin, & Epstein, 1978). TABLE 1 illustrates a representative array of these indicators at the four different time points of intake, during treatment, discharge, and postplacement. At intake, for example, we were interested in the types of youth for whom Boysville constituted a less restrictive treatment environment than either their prior placement or their probable destination if Boysville were not an option. Earlier research by Fitzharris (1985) indicated that for a significant proportion of California youth entering voluntary residential youth care, this was the case. Our own study of agency intake revealed that well over half the youth entering Boysville from the juvenile justice system were either com- ing from or avoiding a more custodial ( i .e . , locked) and restrictive placement (Whit- taker et al . , 1987). We believe this to be a socially significant, albeit partial, indicator of success. Conversely, we had at least an- ecdotal evidence of some youth for whom a residential placement represented a more restrictive, but also more intensive treatment environment ( e .g . , a milieu-oriented drug or alcohol program) indicated for a particular problem or condition. For such youth, the restrictiveness of the residential placement may be offset by the intensity of the treatment offered and thus represent some measure of success at intake as well.

Similarly, during the process of residential

WHITTAKER ET AL 145

__

Table 1 MULTIPLE INDICATORS OF SUCCESS IN RESIDENTIAL TREATMENT

TIME OF MEASUREMENT

Intake

Treatment

Discharge

Postplacement (3-6-1 2-1 8-24-mOS)

CRITERION VARIABLES

Placement in less restrictive environment [or] more restrictiveness offset by level of treatment intensity

Consumer satisfaction Decrease violent acting out Greater family contact than in prior setting Acquisition of prosocial life skills

Planned discharge Discharge to less restrictive setting O 4 of treatment goals attained Consumer satisfaction: youth family, re-

No recidivism Stable school-work placement Community integration-adaptation % treatment goals maintained

ferral agency

POTENTIAL INDICATORS

% of intake diverted from more restric-

% of intake receiving more intensive treat-

Rate of isolation use Rate of assaults Rate of self-injuty Incidence of abuse-neglect Family measures:

tive settings

ment than in prior placement

counseling contact telephone contact family support contact

Life skills: progress from baseline Youth satisfaction Family satisfaction

% completed program %to less restrictive setting % treatment plans completed Referral source satisfaction with place-

% of potential placement days used Number of adjudications O h of treatment goals maintained by level

of supporl in postplacement environment Truancy-work absence rate Continues in less restricting environment

ment outcome

treatment itself, we observed that a number of indicators are both socially desirable in themselves and associated with some desirable future outcome. An example of the former would be a decrease in violent acting out behavior, while the latter includes such things as family contact and support during placement. Again, we were struck by the fact that these were partial indicators of success that we were routinely tracking, though not reporting as such. At the point of discharge, the agency’s initial success criterion, planned termination of residency, was augmented to include planned transfer to a less restrictive setting; this was done be- cause one of our early studies indicated that this composite measure was a more sensi- tive indicator of youth who successfully fin- ished the program (Whittaker, Tripodi, & Grasso, 1987). Other studies have illustrated the utility of a composite measure of percent- age of treatment goals attained at the point of discharge (Nelson et al. 1978), as well as a

variety of “consumer satisfaction” measures during treatment and at the point of discharge (Howard, 1982; Phillips, Phillips, Fixsen, & Wow, 1974). Finally, at various postplacement points, success may be measured in the conventional ways: absence of recidivism, po- lice contacts, stability of school or work, or residential placement. We are presently test- ing a composite measure that includes these indicators and is administered at regular postplacement intervals. While our overall design for evaluation is still tentative, we believe that, ultimately, success at Boysville will be reflected in multiple indicators measured at different time points. Each of the aforementioned indicators represents to us a necessary but not sufficient component of an overall success criterion.

IMPLICATIONS FOR THE FUTURE

Clearly, much remains to be done in spec- ifying an adequate, composite success cri-

146 RESIDENTIAL YOUTH CARE

terion for a complex service like residential youth care and treatment:

1 , At each of the four proposed evaluation points, we encounter the problem of how to measure desired outcomes validly and reli- ably. We must resolve both the multiplicity of available measures for assessing youth, their families, and community variables, and the difficulties involved in adapting those measures to routine data gathering procedures. Sometimes, programmatic innovation re- quires the addition of instruments designed to provide a particular kind of information to clinicians, youth care staff, and program managers. At Boysville, for example, a battery of standardized instruments developed by Mc- Cubbin and associates routinely and repeatedly measures youth and family coping, stress, and adaptation (McCubbin & Thompson, 1987). These data are used to inform and guide an innovative family work program be- gun in the early 1980s. A key factor in the successful completion of such standardized measures appears to be their perceived use- fulness by clinical staff. Overloading the data system with too many instruments can quickly lead to problems of noncompliance and result in critical data gaps. Our experience to date suggests parsimony and caution in adding any new measure to routine data gathering procedures. Our guide, at any point in the treatment process, should be the specific outcome criteria of interest and the particular instru- ment’s ability to predict it. For many standardized instruments there are no norms for the populations typically served by group care agencies-those, for example, overrepresen- ted by single parent and minority families- and may simply add unnecessary complexity to the evaluation process. As a beginning, however, the information yielded by well developed and routinely monitored intake and discharge forms is of considerable value.

2. A related problem concerns the aggre- gation and presentation of data for different audiences: funding bodies, licensing agencies, the media, and professional col- leagues. The multiple indicators discussed

earlier in this paper suggest multiple for- mats for presentation. Boys Town in Ne- braska has developed an interesting format for presentation-a kind of “Dow Jones” index of multiple indicators (Fixsen, Collins, Phillips, & Thomas, 1982). Simi- larly, TABLE 1 might be adapted for an individual agency to provide a summary over- view of performance.

3. For many residential treatment agencies, risk assessment represents an ongoing problem for research. How does one mon- itor the intake queue to determine staff de- ployment and other aspects of resource planning? What clusters of youth-family- environment factors will require excessive use of staff resources? How does one bal- ance the demands of the current resident population with those in intake for an overall assessment of acuity? To a large extent, our existing intake systems are overly weighted towards youth characteristics and underweighted with respect to family and environmental factors, though these latter-availability of postdischarge community support, for example-may be more predictive of community adjustment than individual symptomatology (Whirraker & Pecora, 1984). To the extent that “risks” and “severity” are inextricably tied to funding, this particular measurement issue as- sumes critical importance for residential youth care administrators.

4. As many residential programs move from a child-centered to a family-centered focus, numerous measurement questions are raised (Jenson C? Whittaker, 1987). In our continuing research, for example, we are attempting to identify patterns of continuity with the biological family that predict successful youth outcomes at discharge. We are interested in identifying the components of such involvements: how much contact? oc- curring at what points in the treatment process? what type of family work contact: family treatment? family support? liaison? parent education? and at what level of frequency, intensity, and duration? Similarly, we are in-

WHITTAKER ET AL 147

terested in developing realistic agency goals during the actual placement: elimination of abuse and reduced use of isolation, for example. We are interested in ways of captur- ing individualized differences in results: to illustrate, are there instances where placement in a more restrictive setting could be considered a desirable outcome if that setting offers needed additional treatment?

5. Finally, we are concerned with the organizational requisites for measuring success at any point in the residential treatment process: What types of training and sup- ports are needed for staff? What are com- mon pitfalls and how might these be over- come? How does one insure the accuracy of data? What are the ethical questions involved in continuous evaluation? How are the research and clinical staffs best inte- grated?

Central to our efforts to answer these questions is the availability of routinely and sys- tematically collected data such as those provided by the management information system used in our present research. The opportuni- ties such systems afford for developing valid and reliable indicators for successful youth and family outcomes are exciting.

REFERENCES Banh, R.. & Berry, M. (1987). Outcomes of child

welfare services under permanency planning. So- cial Service Review, 61, 7 1-9 l .

Durkin, R.P., & Durkin. A.B. (1975). Evaluating residential treatment programs for disturbed children. In M. Guttentag & E.L. Struening (Eds.), Hand- book of evaluation research (Vol. 2). Beverly Hills, CA: Sage.

Fitzharris, T. (1985). The foster children of Califor- nia: Projles of 10,OOO children in residential care. Sacramento: Children’s Services Foundation.

Fixsen, D.L.. Collins, L.B., Phillips. E.L., & Thomas, D.L. (1982). Institutional indicators in evaluation: An example from Boys Town. In A.J. McSweeny, W.J. Fremouw. & R.P. Hawkins (Eds.). Practical program evaluation in youth treatment. (pp. 203-230). Springfield, IL: Charles C Thomas.

Gemhenson, C.P. (1956). Residential treatment ofchil- dren: Research problems and possibilities. Social Service Review, 30, 268-275.

Grasso, A.J., Epstein, I., & Tripodi, T. (1987). Agency based research utilization in a residential childcare seffing. Unpublished manuscript, Boysville of Mich- igan, Southfield.

Howard, J.R. (1982). Consumer evaluation of programs for disturbing youth. In A.J. McSweeney, W.J. Fremouw, & R.P. Hawkins (Eds.), Practical program evaluation in youth treatment, (pp. 292-313). Springfield. IL: Charles C Thomas.

Jenson, J.M., & Whittaker, J.K. (1987). Parental in- volvement in children’s residential treatment: From preplacement to aftercare. Children and Youth Ser- vices Review. 9, 81-100.

Jones,R.R., Weinr0tt.M.R.. &Howard, J.R. (1981). Impact of the Teaching Family Model on trouble- some youth: Findings from the National Evalua- tion. Rockville, MD: National Institute of Mental Health (NTIS No. PB82-224353).

Matsushima, J. (1965). Some aspects of defining “success” in residential treatment. Child Weyare, 44 ,

McCubbin, H.I. (1987). Family coping and youth adaptation with residential placement. Unpub- lished manuscript, Boysville of Michigan, South- field.

McCubbin, H.I., & Thompson, A.I. (1987). Family assessment inventories for research and practice. Madison: University of Wisconsin, Family Stress, Coping and Health Project.

Nelson, R.H., Singer, M.J., & Johnsen, L.O. (1978). The application of a residential treatment evaluation model. Child Care Quarterly. 7, 164-175.

Phillips, E.L., Phillips, E.A., Fixsen, D.L., & Wolf, M.M. (1974). The teaching family handbook. Law- rence, KS: Bureau of Child Research, University of Kansas.

Tripodi, T., Fellin, P.A., & Epstein. I. (1978). Dif- ferential social program evaluation. Itasca, IL: Pea- cock.

Whittaker, J.K. (1979). Caring for troubled children: Residential treatment in a community context.San Francisco: Jossey-Bass.

Whittaker, J.K. , Fine, D., & Grasso, A. (1 987). Yourh and family characteristics in residential treatment intake: An exploratory study. Manuscript submitted for publication.

Whittaker, J.K., & Pecora, P. (1984). A research agenda for residential care. In T. Philpot (Ed.), Group care practice (pp. 71-87). Sutton, Surrey, UK: Community Care-Business Press International.

Whittaker, J.K., Tripodi, T., & Grasso, A. (in press). Youth and family characteristics, treatment histo- ries, and service outcomes: Some preliminaxy findings from the Boysville Research Program. Child Welfare.

272-277.

For reprints: lames K . Whittaker. School of Social Work, University of Washington, 4101 15th Avenue, N.E., Seattle, WA 98195

multiple indicators of success in residential youth care and treatment

Documents