insider/partnership evaluation: approach and concept development

Pergamon

Evahation and Program Planning, Vol. 19, No. 3, 199-207, 1996 pp. Copyright 0 1996 Elsevier Science Ltd

Printed in Great Britain. All rights reserved SO149-7189(96)00012-2 0149-7189/96 $15.00+0.00

INSIDER/PARTNERSHIP EVALUATION: APPROACH AND CONCEPT DEVELOPMENT

ALAN FOX, FIONA MCDERMOTT, MARGARET HAMILTON and JOHN TOUMBOUROU

University of Melbourne

ABSTRACT

This paper presents a model of evaluation developed over a period of some 8 years. The aim was to provide a method by which direct service agencies in the mental health and substance abuse areas could cost effectiueiy and appropriately eoaluate their programs. The process adopted was constructed in line with fourth generation ideals alongside an acknowledgement of practitioner wisdom. Our desire was to utilize evaluation not only for performance assessment but also to be an intervention method in itself. Our aim was to integrate evaluation practice in everyday professional activity andpromote a solution to the problems of cross-sectoral service users. Copyright 0 1996 Elsevier Science Ltd

In this paper we will describe the research we have been engaged in during the past 8 years. It has comprised of a process of continual reflection, development and redevelopment as we moved between conceptual work, field research, analysis and reconceptualization. The process led eventually to the interconnection of three research projects Evaluate Yourself (198%89), Not Wel-

come Anywhere (1991-1993) and Project Eve (1995-

1997). We begin the discussion with a consideration of some of the problems that were confronting Project Eve,

we examine the project approach, context and relationships to the other projects, and finally present the concept of evaluation partnerships and evaluation as an intervention activity in itself.

OUR PROBLEM

The main aim of Project Eve was to develop a model of evaluation that was responsive to the needs of drug and alcohol programmes, as well as mental health services programs. In these service areas there are growing pressures on providers to evaluate their services and limited resources to do so. In the current local economic climate cost-effectiveness of an approach was a critical issue. Despite these pressures, practice knowledge and skill about what and how to evaluate in general were often

lacking, particularly in the agency personnel whom the burden for evaluation often falls upon (Schon, 1983; Sechrest, 1994). Models of evaluation adopted from outside an organization’s “culture” had often provided a quick solution. Outsider experts could be contacted by the agency or provider body to provide such evaluation. Such evaluations have a direct high cost in financial terms and resources. The cost could be particularly high when contracting for a fourth generation type evaluation involving many stakeholders in the process. Evaluation utilization in general was also an ongoing problem in the field (Champney, 1993). Without utilization, evaluation would be lacking in cost-benefit provision and again wasteful of scarce resources. The use of multiple roles/perspectives, (Goering & Waylenski, 1993) and the importance of functioning from an internal perspective have also been indicated as important factors in promoting utilization (Champney, 1993) and therefore needed consideration. Adoption of multiple roles would also arguably increase the validity of any evaluation findings. There was, furthermore, the complexity of the evaluative task in health sector service agencies, problematic assessment of outcomes, diversity of approaches and philosophies which combine in the sectors to make the task more complex.

The alcohol/drug and mental health service areas

Requests for reprints should be sent to Fiona McDermott, School of Social Work, University of Melbourne, Parkville, VIC 3052, Australia.

199

200 ALAN FOX et al.

brought the additional problem of cross-sectoral client service delivery into the evaluations. Many service users in Victoria do not just need one service but seemed to be caught between both, often receiving no service at all (McDermott & Pyett, 1993).

Our task then was to develop a model of evaluation which could effectively meet these challenges within the economic realities of modern times. We also wanted to develop a model that tended to instill evaluation as an ongoing process within agencies, to encourage evaluation to become a daily integrated activity, rather than a special one. We sought to formalize what was already done often on an individual haphazard level by service providers already, and include the large array of stakeholders in the process. We considered it most appropriate to use fourth generation/responsive methods. The project described here is our attempt to meet these challenges and is currently collected under one title: Project Eve.

OUR APPROACH

General Background Broadly speaking, we would ‘locate’ our approach to evaluation research within the fourth generation/ responsive approach outlined by Guba and Lincoln (1989), in addition some common ground is shared with developmental models, (Quinn-Patton, 1994), and empowerment evaluation (Stufflebeam, 1994).

Starting their historical review in the second decade of the century, Guba and Lincoln (1987) considered that evaluation was already entering its ‘fourth generation’. Commonly called responsive, these (fourth generation models of evaluation) take as their point of focus not objectives, decisions, effects, or similar organizers but claims, concerns, and issues put forth by members of a variety of stakeholding audiences, that is, audiences who are in some sense involved with the evaluation “and hence are put at risk by the evaluation” (Cuba & Lincoln, 1987).

The principles undergirding these responsive models may be noted to a greater or lesser degree in responsive evaluation, naturalistic evaluation (Guba & Lincoln, 1987) illuminative evaluation (Gordon, 1991), utilization focused evaluation (Patton, 1986) and adver- sarial evaluation (Guba & Lincoln, 1987).

Development Context of Project Eve Between 1988 and 1992 a method for evaluating treatment in drug and alcohol services was developed, result- ing in the publication of an evaluation handbook Evaluate Yourself (McDermott, Pyett & Hamilton, 1992). This research project required extensive con- sultation with agencies in that sector in three Australian states; Victoria, New South Wales and Western

Australia. The method devised was trialed broadly in those States.

Evaluate Yourself was a method which relied on the development and testing of those outcome variables relevant to the ‘local conditions’ within each agency or programme. It was a collaborative approach in which service users, providers and many of those who have a stake in the outcome of a program, work together to explore, understand and measure treatments or inter- ventions in relation to their specific program goals and aims. Its effectiveness as a method of organization development as well as evaluation has been commonly reported by those who have used it (see McDermott & Pyett, 1992 for a discussion of related theoretical and ethical issues).

Experience with the Evaluate Yourself handbook led its authors to conclude that this intervention produced an increase in evaluation activity. Even in agencies where evaluations were incomplete, there were observed increases with respect to understanding and acceptance of evaluation (Hamilton, McDermott, Lagay & Farrow, 1989). Such advantages were considered to have pos- sibly been enhanced in organizations that combined the use of Evaluate Yourselfmaterial with additional forms of support, such as outside facilitators or evaluation resource groups (Pyett, McDermott, Hamilton & Robinson, 1992).

Experience with Evaluate Yourself also suggested to its authors (McDermott, Pyett & Hamilton, 1992), that the work involved in examining and considering the self-evaluation process may have had positive organizational advantages. While there was evidence of such benefits being associated with self-evaluation participation, the proposition was in need of further research.

However, a significant finding from these trials was a contradiction: “while service providers valued and endorsed the legitimacy of this self-evaluation method they seemed unable to fully adopt it without the resource of outside consultants” (Keenan, 1993).

This then became our dilemma: how do we maximize the benefits of “insider” evaluation (that is, evaluation which “stays with” and builds upon the knowledge and expertise of service providers and users) without resort- ing to “outsider” resources (and the consequent loss of autonomy and ownership over the evaluation) to carry it through?

The structure r b ihe project subsequently needed re- thinking and modification. Changing social context, academic discourse development, and burgeoning economic rationalism in the government sector all con- tributed to a need to expand and re-design earlier project features.

Project Eve was initiated to meet these changes and also to include an expanded interest of the research team towards evaluation in the mental health sector and

Insider/Partnership Evaluation 201

the problems of clients with cross-sectoral needs in both sectors.

Project Eve Project Eve sought to further develop the evaluation method termed Evaluate Yourself so its utility and effectiveness would be improved in line with the changing needs of the drug and alcohol sector as well as the needs of the growing mental health sector following de- institutionalization. The original handbook has been updated (Fox, McDermott, Hamilton & Toumbourou, 1996) and adaptations made to the method so that cross- sectoral utility can be promoted.

Project Eve then, built on the original materials and sought also to deal with the following.

1. The pairing of problems- serious psychiatric disorder and problematic drug and alcohol use-and the consequences of poor partnerships between service systems addressing each problem separately; and

2. the pairing of two initially separate research projects: Evaluate Yourself and Not Welcome Anywhere

Paired Problems The literature examining the co-occurrence of psychiatric illness and problematic drug or alcohol use is characterized by a lack of agreement over definitions and a consequent absence of a satisfactory classification system (Attia, 1988; el-Guebaly, 1990; Ford & Kamerow, 1990; Ford, Hillard, Geisler & Lassen, 1989; Frances, 1988; Kofoed, 1991; Kosten & Kleber, 1988; Osher, 1989; Rivinus, 1988; Ryglewicz, 1991; U.S. Department of Health & Human Services, 1991; White & White, 1989; WHO, 1991). One complicating factor, as several writers note (Brown, Ridgely, Pepper, Levine & Ryglewicz, 1989; Lehman, Myers & Corty, 1989; Ryglewicz, 1991) is the interactive aspect of these conditions that places mentally ill persons at high risk of problematic drug or alcohol use, and those with problematic drug or alcohol use at high risk for mental illness.

The nature and direction of the relationship between psychiatric disorder and problematic drug or alcohol use, and the potential for this association to be harmful has also been the subject of continuous and inconclusive debate (Casteneda, Galanter & France, 1989; el-Geub- aly, 1990; Weller, Ang, Latimer-Sayer & Zachary, 1988; WHO, 1991).

Despite these conceptual and definitional difficulties that clearly reflect the complexity of the issue, many researchers agree that considerable numbers of people commonly experience these dual problems (Bauer, 1987; Kiesler, Simkins & Morton, 1991; WHO, 1991). In the U.S. it has been observed that people with these dual problems are increasingly becoming the majority of the population in need of mental health services (el-Gueb-

aly, 1990; Johnson, 1990; Lett, 1988; Ponce & Jo, 1990; Rosenheck, Massari, Astrackan & Suchinski, 1990; Ryglewicz, 199 1).

The problems posed to service providers, patients, families, carers and the community in general by the pairing of these problems are complex and considerable. They include: enormous personal pain; loss and rejec- tion; premature death; sometimes imprisonment; home- lessness; a sense of powerlessness and frustration in carers and service providers.

In 1991-1992 we embarked on a study of the situation of people in Victoria who experience serious psychiatric disorders and problematic drug or alcohol use (McDer- mott & Pyett, 1993).

Our study was within an action research framework in which we relied upon participation, collaboration, dialogue and feedback amongst those most affected by this issue. To this end we worked with service providers, administrators and policy makers on a cross-sectoral basis, including those in the mental health, homeless, and drug and alcohol treatment sectors. We also attempted to engage with as many people as possible who themselves experience these dual problems.

The findings drawn from the interpretation of data arising out of this multi-method, cross-sectoral study enabled us to engage with the complexity surrounding co-occurring serious psychiatric disorder and problematic drug or alcohol use. At the same time we were able to identify some areas of shared ground amongst those who confront these issues.

There were three key aspects of our findings: these refer to:

l The prevalence of the problem; l service provision; and l problems and dilemmas.

Prevalence. There currently exists no statistical data in Australia to establish the numbers of people experi- encing these dual problems.

We conducted a survey of almost 600 mental health, housing and drug and alcohol agencies in Victoria in which we asked the respondents (amongst other questions) to estimate the numbers of people with these dual problems who made contact with their agency on an average month. Our calculations were based on the estimates provided by 417 respondents (a response rate of 72%).

Respondents indicated that between 487 and 880 people who have both a serious psychiatric disorder and problematic drug or alcohol use were contacting Vic- torian agencies each week, Even if these figures include a number of repeat contacts they represent the extent of the current call on existing services.

202 ALAN FOX et al.

Service Provision. Of the 322 Victorian agencies contacted by this client group, almost 300 agencies indicated that they were providing some services to these clients. However, most agencies noted that the service they provided was assessment or referral. Calculations derived from these responses indicated that between 195 and 478 people with these dual problems were being referred each week to other agencies. This may mean that they were being referred for assistance for their “other” problems, or for accommodation, health or welfare needs.

Comparing the numbers of people with both a serious psychiatric disorder and problematic drug or alcohol use who contact agencies each week, and the numbers of these clients who were using services each week in the same agencies, it was apparent that there was a substantial gap across all sectors. According to these estimates, between 200 and 300 people with both a serious psychiatric disorder and problematic drug or alcohol use were contacting Victorian agencies each week and receiving no service whatsoever.

Problems and Dilemmas. In reflecting on the data it became evident that there were a number of factors which impede the provision of services to this client group. In particular we shall discuss two of these:

l Administrative divisions, and l cross-sectoral philosophical differences.

Administrative Divisions. The mental health and drug and alcohol treatment sectors in Victoria have, in recent years, developed separately, providing separate services and requiring separate training for service providers. Further, people with both serious psychiatric disorders and problematic drug or alcohol use are frequently excluded from one or the other service. Indeed it has been argued that the “dually disabled” are in an important sense created by these administrative divisions (Lehman, Myers & Corty, 1989; Osher, 1989).

However recent policy developments evident in the 1992 Victoria Drug Strategy and the 1992 National Mental Health Policy and Plan indicate a growing awareness of the necessity to move towards an integrated health service system. This is a difficult task and, as we shall discuss below, in the case of persons with these dual problems there are significant philosophical and treatment issues to be addressed.

Cross-sectoral Philosophical Differences. In examining the philosophies underlying conceptualizations of mental health and alcohol and drug problems, it is apparent that differences exist that impact directly upon the way in which people who experience these dual problems are

assessed and treated or managed (el-Guebaly, 1990). Our research pointed to a number of contradictions:

l Decisions made in the mental health sector about the optimal use of medication for psychiatric disorders contrast with the fear in the alcohol and drug sector of fostering (further) chemical dependency; and

l the widespread belief in the mental health sector that the person with a psychiatric disorder needs encour- agement and support to participate in a programme contrasts with the emphasis on motivation as a pre- requisite for treatment for most alcohol and drug problems.

At the same time, service providers engaging with people with these dual problems are faced with the following dilemmas:

l Whether abstinence should be a condition or even a goal of treatment;

l whether abstinence can be a condition for treatment while compliance with medication is also required;

l the difficulty and lack of knowledge concerning the nature of the casual relationship between drug or alcohol use and psychiatric disorders;

l the problem of who should treat what first, or if both should be treated concurrently, and by whom; and

l the lack of knowledge about which treatment or management methods from each area of specialization are the most effective, not only for those with one problem, but also for those with dual or multiple problems.

A further contradiction emerged out of our dis- cussions with people who themselves experience these dual problems. Many consumers with serious psychiatric disorders consider their alcohol or drug usage to be a solution to their difficulties, offering them some symptom relief, the possibility of a social group and the means whereby the dampening effects of their pre- scribed medication could be counteracted.

Throughout the 2 years of this research we enlisted the participation of service providers, service users and consultants in many sectors, government and non- government. This process was in itself valuable, providing an opportunity for many areas of commonality and difference to be debated in the workshops and sem- inars that were organized. It reinforced our belief that there is a pressing need for the development of shared philosophical ground amongst service providers across all sectors, so that the range of treatments and inter- ventions provided do not perpetuate administrative divisions, and the consequent inability for an integrated network or services to be established and maintained.

Paired Projects Our reflections while working on Not Welcome Any- where drew us back to the work on Evaluate Yourself and the potential for finding some solution to both


projects outcomes and problems by combining aspects in the new project: Project Eve. Evaluation methods and models were needed that included cross-system measures in order to understand the impact of services on the outcome of the client’s functioning (Burns, Ang- old & Costello, 1992; Morrissey, 1992). For Australia this was a new and innovative approach to evaluation, which was also consistent with current policies in the health and welfare fields. It was hoped that this research itself would play a role in establishing cross-sectoral dialogue, and the sharing of treatment and intervention philosophies, practices, outcome expectations and measures consistent with an integrated view of the nature of human behavior.

The development of theory from practice has, for the project as a whole, followed the processes and tenets of Grounded Theory (Glaser & Strauss, 1967). Systematic data collection and ongoing analysis combine to con- stantly compare developing theory with feedback from the ongoing data. This spiral development of theory and practice is illustrated in Figure 1. The grounding of our theory in practice has been the hallmark of its development. Multiple perspectives of participants were critical in this process if interpretations were to have any context validity.

There is also the need to include the notion of reflexivity, the idea that evaluation should generate the type of data that can be turned back on the participants, assisting, locating and grounding them as evaluators within the evaluation context (Steier, 1991). The reflexivity notion undergirds both the development of our conceptualization of evaluation and the practice of it.

Several further assumptions underlay our approach; we considered that there was particular value in acknowledging and utilizing agency practitioner’s wisdom with respect to programs and clients. As a resource these are the group who have some of the most spe- cialized knowledge to assist in both the interpretation of evaluation data and the promotion of evaluation utilization within an agency.

There were considerable “spin offs” and long-term

Figure 1. Project review and development context.

commitments to be gained through the use of collaborative approaches. They would become evident in relationships, patterns of communication and assist in the creation of user-friendly networks.

We believed that multiple methods would give a more coherent, valid, picture of events in a way which would be hard to replicate using single method designs.’ Use of any one method alone was unlikely to promote an analysis that would capture the complexity of the mean- ing of any program and its outcomes have service users in the contexts of drug and aIcoho1 services or mental health services. The more points of view that were promoted, the more likely it was that consensus over an analysis could be promoted with all stakeholders. The more likely findings would be accepted and utilized as valid within that context.

A further premise was that the agency personnel had a right to “own” the evaluation and the findings. If the agency personnel do not feel personally part of the activity, or listened to appropriately, it is unlikely they would learn anything very useful from the process. They had to be a major part of the process if evaluation was to become an integrated activity within the agency, and utilization promoted as part of ongoing development.

In addition, we asserted that there was value in involving clients and other stakeholders in many aspects of agency program evaluation and what it means to them. If the service users perspectives along with other stakeholders were not sought or integrated into any interpretation then that interpretation may be less trust- worthy and would be harder for all parties to accept and learn from.

The process or method of evaluation which we had been developing had arisen inevitably from our epistemology- which is to say that the methodology is synonymous with the epistemology. Very importantly, it has been a method which relies upon close collaboration with agency personnel and, to a varying extent, with clients. Our concern to develop a “user- friendly” process had led to a method that agency staff can make use of and control themselves, giving them the option of delegating tasks to outside consultants or others to a greater or lesser extent as their needs define. The morphology of the method is such that the pec- uliarities and idiosyncrasies of different settings can be evoked and explored using data gathering methods that were “natural” to particular treatment contexts. Thus

‘Recent trends in programme evaluation in general (Smith, 1994) seem to indicate that interest for many in the quantitative/qualitative debate is waning with respect to the issue of superiority except for

those who are continuing to argue on the epistemological level (Light, 1994). Most evaluation writers now seem to agree that quantitative

and qualitative methods both have meaningful contributions to make,

and that the dual/integrated usage of the methods probably allow a fuller picture of events to be established (Bickman, 1994; Boruch,

1994).

204 ALAN FOX et al.

the possible range of suitable data collecting methods was broad, the intention being to find and “open” as many “windows” as possible onto the treatment matrix. This also means that it is possible to replicate the process and to make comparisons of findings across settings while still capturing the uniqueness of individual settings.

One of the particular contributions of this type of evaluation is that it can take place within and be absorbed into the ever-changing and uncertain reality that is synonymous with human action. In this sense, evaluation can “start where the client is”, acknowledging and making use of the very factors (change, turbulence, conflict) which are sometimes seen as inimi- cal to evaluation yet indivisible from the everyday realities of practice.

Partnerships In bringing together the findings from these two orig- inally unconnected projects we were struck by several points of convergence:

l The negative consequences for service delivery from a fragmented and philosophically separated service system;

l the efficacy of evaluation as a form of organization development and change;

l the potential of constructive cross-sectoral dialogue to benefit service users and service providers in the drug and alcohol, homeless, and psychiatric disability sectors; and

l the need for further conceptualization and model development to increase self-evaluation activity and maximize its potential.

Recognizing this convergence we decided to utilize it within Project Eve and set ourselves two further objectives:

1.

2.

To address the insider-outsider contradiction by seeking ways to increase self-evaluation practices; and to use evaluation as an intervention to the extent that, as a practice, it might create a degree of elas- ticity in agency boundaries through information sharing and increased understanding of internal agency philosophies and intervention strategies.

To achieve both these purposes Project Eve has as its focus the development of evaluation partnerships. That is, we propose to form evaluation teams between agencies in each sector: a drug and alcohol treatment agency partners a mental health agency. These agencies will be located in the same region and will be working with some of the same clients, for example service users with the dual problems of serious psychiatric disorders and problematic drug and alcohol use. Evaluation partners will receive training from the project team in using a

revised and updated edition of Evaluate Yourself handbook (Fox et al., 1996) and will meet regularly to guide, assist and support one another’s evaluative work using

this method. Thus, evaluation partners will be “insiders” in their

own agencies and “outsiders” in their partner’s agency. However, unlike outside consultants they will have a much closer understanding of agency “life”, its vicissi- tudes, pressures and problems.

Simultaneously they will have a degree of distance and objectivity to bring to their work with their partners. We anticipate that this “outside” perspective on the “inside” of another’s work environment will have benefits in sensitizing them to what they might take-for- granted or non-problematize within their own setting.

We envisage several benefits accruing from such a positioning. Firstly there is the possibility of improved validity of evaluation findings. When internal evaluations are used for external reporting there is always the potential that biasing of the findings will be motiv- ated due to questions of retaining funding in a com- petitive environment. While this is not necessarily the case, the use of a peer supervisory partnership relation should well reduce the possibility of such an occurrence.

Secondly it is hoped that improved evaluation utilization within agencies will be promoted. Earlier work with Evaluate Yourself seems to support this and the inter-agency relationship may promote an atmosphere of friendly competition that supports the integration and use of evaluation findings.

A partnership approach is a more cost effective use of resources than outside expert employment. It also has the benefit of increasing within agency skills and developing a within agency integration of evaluation practice. Ideally it will make evaluation a normal part of everyday professional work on a formal level.

The use of partnerships should lead to improved cross-sectoral communication and resource allocation as it sets a path for communication with common services on the agenda. As agencies will develop knowledge of cross-sectoral service users, it will no longer be so easy for such people to be lost between the services and greater agency, accountability for those services or new ones should be promoted.

The development of improved cross-sectoral understanding may form the basis for other joint activities. A door will be opened to this opportunity.

DISCUSSION

Our purpose overall has been to explore and utilize the self-other relationship between agencies operating at a similar service provision “level”. This differs from the more oppositional character evident in the evaluation consultant-service provider position that to a degree,


could be said to have occurred in our earlier work on Evaluate Yourself. While service providers requested “more of the same”, believing that outside experts could ensure the carrying through of an evaluation, our belief is that the self-other relationship has not been explored. Its potential for sustaining useful evaluation practices and for forging linkages amongst players in the treatment network which can work to the benefit of both service providers and users requires attention.

In earlier work our group has assisted self-help groups in the drug and alcohol area in analysing and reporting the results of their evaluation of the formal service delivery system. Experience with this project suggested to us that research partnerships could be of value in encouraging reflection and critical analysis in the semi-formal service sector (Toumbourou, Hamilton & Smith, 1994).

These considerations have become more pressing in the drug and alcohol treatment sector in recent years. In Australia, the philosophy underpinning drug and alcohol service delivery has steadily shifted, with a pub- lic health harm-reduction framework gaining in credi- bility. These changes have increased the perception of differences within some of the traditional 12-step groups, specifically Alcoholics Anonymous and its derivatives such as Narcotics Anonymous (Toum- bourou & Hamilton, 1994).

We have approached these differences by emphas- izing research partnerships as a potentially valuable method for self-help group members to explore their own practices. By encouraging involvement in research publication, self-help members have increased oppor- tunities for important components of the self-help approach to receive research attention (Toumbourou & Hamilton, 1994). For example the notion of “spiri- tuality” that represents important values for self-help members, are yet nowhere acknowledged in the emerg- ing harm-reduction discourse.

These developments support the potential of evaluation partnerships as arrangements for encouraging communication and critical reflection across diverse and different service sectors.

The two sectors we are working with have tra- ditionally had problems developing adequate protocols for working with each other on referral and other service levels. Various protocols have been attempted in Vic- toria (e.g. Department of Health & Community Services, 1995) but have stumbled over the complexity and philosophical problems of developing such formal relationships. The work of Lisbeth Schorr, Sid Gardner and the National Centre for Service Integration in the U.S.A. on institutions and agency service integration, show the benefits for a more integrated approach to service provision in the education and welfare sectors (Gardner 1993, 1994; Schorr, 1992). However this process has the backing of a national movement towards

such ends. In the sectors we have chosen in Victoria the backing for such moves is less developed and problematic for reasons previously outlined. It is hoped that starting the service integration process, at the level of promoting cross-sectoral understanding through evaluation, that other barriers to the process of developing integrated appropriate services will be weakened.

The preliminary work that we have done so far in Project Eve suggests that agencies will welcome the opportunity to participate in this research. Our initial recruiting and training of evaluation partners is sched- uled for early 1996, and we are optimistic that this partnership approach will lead to the benefits envisaged.

We are aware of possible problems that may be enco- untered, for example; poorly chosen staff in terms of personality, skills or abilities to carry out this complex work; or insufficient organizational or sectoral support being given in financial or other resource time to facilitate changes and support the worker, etc. Careful selec- tion of appropriate individuals and negotiation with staff and management structures using a systems perspective will overcome some of these difficulties. Chan- ges in legislation with respect to evaluation in government supported agencies and competition for funding, mean that organizational support and resource allocation for the evaluation activity are likely to be available. The climate is right in Victoria for a cost- effective and stakeholder focussed process such as we are proposing.

We have been encouraged by the outcomes from work in the self-help area previously referred to (Toum- bourou & Hamilton, 1994). We believe that Project Eve will allow us to gain a greater understanding of the kinds of conditions that facilitate or inhibit partnership work, the different ways in which partnerships evolve, and the different types of partnerships that may be possible. Very importantly, we believe that this research will enhance evaluative activity, the creation of evaluation cultures within and amongst agencies, and the utilization of evaluation findings. During 1996 and early 1997 field testing of these propositions is being carried out.

REFERENCES

ATTIA, P.R. (1988). Dual diagnosis: definitions and treatment.

Alcoholism Treatment Quarterly, 5, 53-63.

BAUER, A. (1987). Dual diagnosis patients: the state of the problem, T.I.E. Lines, 4, 1-8.

BICKMAN. L. (1994). An optimistic view of evaluation. Evaluation Practice, 15(3), 255-259.

BORUCH, R.F. (1994). The future of controlled randomised experi- ments: A briefing. Evaluation Practice, 15(3), 265-274.

206 ALAN FOX et al.

BROWN, V.B., RIDGELY, MS., PEPPER, B., LEVINE, IS., (1989). Starting where the client is: evaluation of treatment programs,

RYGLEWICZ, H. (1989). The dual crisis: mental illness and sub- Proceedings of the 21st National Conference of the Australia Associ- stance abuse. American Psychologist, 44, 565-569. ation of Social Workers, Townsville, July, 156157.

BURNS, B.J., ANGOLD, A. &COSTELLO E.J. (1992). Measuring

child, adolescent and family service use. In L. Bickman & D. Rog,

(Eds), Evaluating Menial Health Services for Children, No. 54, San

Francisco: Jossey-Bass.

JOHNSON, B. (1990). Substance mis-use among psychiatric hospital

patients. Irish Journal of Psychological Medicine, 7, 114-l 11.

KEENAN, M. (1993). Final report of the VAADA evaluation project. Melbourne: VAADA.

CASTENEDA, R., GALANTER, M., FRANCO, H. (1989). Self-

medication amongst addicts with primary psychiatric disorders. Com-

prehensive Psychiatry, 30, 8&83. KIESLER, C.A., SIMKINS, C.G., MORTON, T.C. (1991). Preva-

lence of dual diagnosis of mental and substance abuse disorders in

general hospitals. Hospital and Community Psychiatry, 42,40@-403. CHAMPNEY, T.F. (1993).Comments on the use of multiple roles to

maximise the utilisation of outcome research. Evaluation and Program Planning, 16, 335-336.

KOFOED, L. (1991). Assessment of co-morbid psychiatric illness and substance disorders. New Directions for Mental Health Services, 50, 43-55.

D.H.8tC.S. (1995). Protocol between psychiatric services and drug and alcohol services. Melbourne: D.H.&C.S.

EL-GUEBALY, N. (1990). Substance abuse and mental disorders:

the dual diagnosis concept, Canadian Journal of Psychiatry, 35, 261- 261.

KOSTEN, T.R. & KLEBER, H.D. (1988). Differential diagnosis of

psychiatric co-morbidity in substance abusers, Journal of Substance Abuse Treatment, 5, 201-206.

FORD, D.E. & KAMEROW, D.B. (1990). Screening for psychiatric

and substance abuse disorders in clinical practice, Journal of General International Medicine, 5, 53141.

LEHMAN, A.F., MYERS, C.P., CORTY, E. (1989). Assessment

and classification of patients with psychiatric and substance abuse

syndromes. Hospitaland Community Psychiatry, 40, 1019-1025.

LETT, P. (1988). Dual diagnosis. Journal of Applied Rehabilitation Counseiling, 19, 1620.

FORD, J., HILLARD, J.R., GEISLER, L.J. & LASSEN, L.K.

(1989). Substance abuse/mental illness, American Journal of Drug and Alcohol Abuse, IS, 297-307.

LIGHT, R.J. (1994). The future of evaluation. Evaluation Practice, 15(3), 249-253.

FOX, A., MCDERMOTT, F., HAMILTON, M. Kc TOUM-

BOUROLJ, J. (1996). Evaluate Yourse!f(2nd Revised Ed.), Canberra: AGPS.

MCDERMOTT, F. & PYETT, P. (1992). Minimising harm in

research. International Journal of Drug Policy, 3, 3.

FRANCES, R.J. (1988). Update on alcohol and drug disorder treat-

ment. Journal of Clinical Psychiatry, 49, (suppl.). 13-17.

MCDERMOTT, F. & PYETT, P. (1993). Not Welcome Anywhere. Vol 1 & 2. Melbourne: VICSERV.

GARDNER, S. (1993). Afterword. Journal of Education Policy, 6, 5- 6, 189-199.

MCDERMOTT, F. & PYETT, P. (1994). Co-existent psychiatric

illness and drug abuse: A community study. Psychiatry, Psychology and Law, l(l), 45-52.

GARDNER, S. (1994). Training for the future: Family support and

school-linked services. Equity and Choice, 10(3), 16-18

MCDERMOTT, F., PYETT, P., HAMILTON, M. (1992). Evaluate Yourself, Canberra: AGPS.

GIDDENS, A. (1979). Central problems in social theory. Berkeley,

CA: The Macmillan Press.

GLASER, B. & STRAUSS, A. (1967). The discovery of grounded theory: strategiesfor quantitative research. Chicago: Aldine.

MORRISSEY, J.P. (1992). An interorganisational network approach

to evaluating children’s mental health service systems. In L. Bickman

& D. Rog (Eds), Evaluating mental health servicesfor children, No 54. San Francisco: Jossey-Bass.

GOERING, P.N. & WAYLENSKI, D.A. (1993). Promoting the

utilisation of outcome study results by assuming multiple roles within

an organisation. Evaluation and Program Planning, 16, 329-324.

OSHER, F. (1989). The dually diagnosed: patient characteristics

and treatment strategies, Community Support Network News, June- July.

GORDON, K.H. (1991). Improving practice through illuminative evaluation. Social Service Review, September, 36.5-378.

PATTON, M.Q. (1986). How to use qualitative methods in evaluation. New York: Sage.

GUBA, E.G. &LINCOLN, Y.S. (1987). The countenances of fourth- generation evaluation: Description, judgement and negotiation. In

D.S. Cordroy & M.W. Lipsey (Eds), Evaluation Studies Review Annual, Vol. 11 Newbury Park, CA: Sage.

PONCE, D.E. & JO, H.S. (1990). Substance abuse and psychiatric disorders. Residential Treatment for Children and Youth, 8, 5-15.

GUBA, E.G. & LINCOLN, Y.S. (1989). Fourth Generation Eualu- afion, Newbury Park, CA: Sage.

PYETT, P., MCDERMOTT, F., HAMILTON, H. & ROBINSON, R. (1992). Evaluation in action: Making it work in the drug and alcohol field. Proceedings of Australasian Evaluation Society Conference 1992.

HAMILTON, M., MCDERMOTT, F., LAGAY, B. & FARROW, J.

QUIN-PATTON, M. (1994). Developmental evaluation. Evaluation Practice, 15(3), 31 l-319.


RIVINUS, T.M. (1988). Difficulties of diagnosis, differential diag-

nosis and dual diagnosis in the late adolescent and young adult sub-

stance abusers. Journal of College Student Psychotherapy, 2, 115- 120.

ROSENHECK, R., MASSARI, L., ASTRACKAN, B. & SUCHIN- SKI, R. (1990). Mentally ill chemical abusers discharged from VA

inpatient treatment. Psychiatric Quarterly, 61, 400-403.

RYGLEWICZ, H. (1991). Psychoeducation for clients and families.

Psychosocial Rehabilitation Journal, 15, 79-92.

SCHON, D. (1983). The reflective practitioner, New York: Basic

Books.

SCHORR, L. (1992). Commentary: Reason to hope. Teachers College Record, 93(4), 71&716.

SECHREST, L. (1994). Program evaluation: Oh what it seemed to

be! Evaluation Practice, 15(3), 359-365.

SMITH, M.F. (1994). Evaluation: Review of the past, preview of the

future. Evaluation Practice, I5(3). 215-227.

SOBELL, M.B., BROCHU, S., SOBELL, L.C., ROY, J. & STEV-

ENS, J.A. (1987). Alcohol treatment outcome evaluation meth-

odology: State of the art 1980-84. Addictive Behaviours, 12, 113- 128.

STEIER, F. (1991). (Ed.), Research and reflexivity. London: Sage.

STUFFLEBEAM, D.L. (1994). Empowerment evaluation, objec-

tivist evaluation, and evaluation standards: Where the future of evalu-

ation should not go and where it needs to go. Evaluation Practice, 15(3), 321-338.

TOUMBOUROU, J. & HAMILTON, M. (1994). Researching self-

help drug treatment: collaboration and conflict in the age of harm

reduction. Addiction, 89(Z), 151-156.

TOUMBOUROU, J., HAMILTON, M. & SMITH, R. (1994). Sur-

veying the drug service users’ perspective through self-help groups,

Journal of Community and Applied Social Psychology, 4(2), 13 l-140.

U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES

(1991). Drug abuse and drug abuse research, 3rd Trienniel Report to

Congress. U.S.A.

WELLER, M.P., ANG, P.C., LATIMER-SAYER, D.Y. & ZACH- ARY, A. (1988). Drug abuse and mental illness. Lancet, 2, 977.

WHITE, K.K. & WHITE, D.E. (1989). Dual mental health and

substance use problems: a model of four subtypes, Psychosocial Rehabilitation Journal, 13, 93-98.

WORLD HEALTH ORGANIZATION. (1991). Bulletin, 59, 225-

242.

insider/partnership evaluation: approach and concept development

Documents