Building and Maintaining Consumer Organizations:

Using Evaluation to Support Consumer Delivered Services

Cindy WilleyWashington Institute

for Mental Illness Training and Researchckw4@u.washington.edu

Eric J. BrunsUniversity of Washington

Division of Public Behavioral Health and Justice Policyebruns@u.washington.edu

Description of the session Introductions

Why did we attend this session? Setting the stage

A brief overview of research and evaluation Why is it relevant for consumers and families?

Consumer-led evaluations History and examples

Brainstorming and group discussion Evaluating your program in Washington State

Getting Ideas and Feedback The Washington State Consumer and Family

Evaluation Mini-Grant Program

Consumers, family members, and researchResearch has led to many breakthroughs

in treatment and rehabilitationSuch breakthroughs are only possible

through teamworkSo why is research sometimes not seen as

relevant? Research as a mystery Research as something consumers don’t

participate in Research as something not relevant to everyday

life

Research Questions from consumers What factors help people to recover?

Or, what factors hinder recovery? What kinds of jobs supports can help people with

psychiatric disabilities succeed? What factors make treatment most useful and

meaningful? How does spirituality affect recovery? What is the relationship between mental health and

physical health? How do budget cuts affect the mental health of

consumer/survivors? What is the impact of involuntary commitment on

someone’s mental health? What are the benefits or risks of taking a specific

medication?

Program Evaluation A growing emphasis on outcomes

(Outcomes = Documented result or impact of a program, intervention, or process…)

A positive development for consumers? All programs will have to show how services helped to

improve the lives of their clients (i.e., not just how many people they served or how many

units of service were provided) A negative development for consumers?

For consumer-run programs, more responsibility for conducting research or evaluation that shows whether these programs are beneficial to their participants.

How to make research and evaluation more relevant More useful products – “real-world” information focused

on daily living (employment, transportation, housing) Research reports in more simple language and in varied

formats that are user- friendly Better understanding by researchers of cultural

differences and needs. Recruit, train and hire more researchers with disabilities

and from minority backgrounds. More public education should be provided. Consumers should be full participants in planning and

doing research Build greater trust between researchers and consumers.

From the National Center for the Dissemination of Disability Research and Research Utilization Support and Help (RUSH) project

Evaluation Methodology, Design and Analysis:

An Overview

Process and Outcome Evaluation Process Evaluation (a.k.a. implementation assessment, formative

evaluation, intervention fidelity) Is the program being delivered as intended? Is the program being delivered to who it was intended?

Outcome Evaluation (a.k.a. impact assessment, summative evaluation) What is the effect of the program on the program recipients? Is the program beneficial? For whom is the program beneficial? More for some than

others?

Process evaluation examplesConsumer satisfactionConsumer-oriented valuesService delivery and practice

Availability General assessment of service quality Treatment quality / treatment fidelity

System development Service coordination and collaboration Assessment of community capacity

Why process evaluation?Testing assumptions in an evaluation of respite care

Information about respite availability and respite’s benefits will reach the target audience

When parents and providers hear of the program, they will refer/be referred to the program

Respite workers will be well-trained to deal with problem behaviors Respite workers will be able to offer activities during respite hours that

are attractive and beneficial to the child and the parent Parents will be able to use allotted respite care time in a way that

reduces stress Respite will be of adequate intensity and duration to be of benefit to

children and parents Parents and children will perceive the respite care program to be helpful Respite provides a “value-added” to other services and supports

Outcomes Evaluation:Experimental Evaluation Design

Random assignment into intervention and control groups

Group comparabilityMinimize threats to internal validity

• Internal validity: Are the outcomes associated with the intervention; Can we say that the intervention causes the outcome?

Can only be used to evaluate programs that do not target entire populations

Experimental Designs: The Realities

Offer best assessment of cause and effect

Offer best linkage between program and outcomes

The “Gold Standard” in determining “Evidence Based Treatments”

May not be ethical May not be legal May not be feasible

Examples of Quasi-Experimental Designs (when program does NOT target entire population)

Post-test only comparison group designs Two groups (intervention and comparison) observed following the

presentation of intervention No random assignment to groups No pre-testing

Pre-test post-test comparison group designs This is the most frequently used design in social science evaluation

research. Involves a pre and post intervention measurement for the two groups (intervention and comparison), no random assignment to groups.

Examples of Quasi-Experimental Designs (can be used when program targets entire population)

Repeated measures designs Can be used with just the intervention group or with two groups

(intervention and comparison). Several measurements over time (e.g., baseline, post intervention, one year subsequent, etc.)

Time series designs Expanded repeated measures, include many observations of

either just the intervention group or two groups (intervention and comparison)

Intervention may be introduced and removed and reintroduced At least 50 observations points required

Data Collection Methods Self-report: Individuals report/describe their own

behavior/state of mind/feelings/etc. OR some other informant provides information about the

individual of interest (e.g, parent reports on child behavior)

Observation: Researcher observes and records (in some way) the behavior of interest.

Archival Data/Existing Records/MIS

Evaluation Principles to considerTheory-based

What are the expected outcomes? Why do we think we will get them?

Utilization-focused What are the priorities for our evaluation resources?

• Participatory• How do we make sure the evaluation is informed and used by all the

relevant stakeholders?

Theory-based: Clubhouse model

Clubhouse

This example courtesy of Dori Hutchinson, USPRA Research Committee

Utilization focused:Asking questions about “What to measure?”

Types of questions: Evaluation use What decisions, if any are the evaluation’s

findings expected to influence? When will the decisions be made? By whom? When, then, must the evaluation findings be

presented to be timely and influential?

Utilization focused:Asking questions about “What to measure?”

Types of questions: What data is needed? What data and findings are needed to support

decision-making? Are the data available through existing

means? …Or, are new data collection approaches

needed?

Participatory Evaluation Broadly identify stakeholders and include more non-

evaluators than evaluators in the evaluation (e.g., consumers, families, youth, program directors, community groups, etc.). A “stakeholder” = Anyone who has a stake in

the program that is being evaluated!! The group of stakeholders of a program and its evaluation may

not be static. Regularly update stakeholder groups on the data collection

process. Learn how it can be improved, refined, enhanced. Convene stakeholder group to design data dissemination plan. Disseminate results regularly to and with stakeholders

HISTORY AND EXAMPLES OF CONSUMER-LED EVALUATION

CINDY WILLEYWASHINGTON INSTITUTE

In 1979 Prager and Tanaka reported to the Ohio Department of Mental Health on the results of involving mental health consumers in evaluation. They concluded: “Representing the consumer’s perspective on the meaning of mental illness and the correlates of ‘getting better,’ the process of client involvement in evaluation design and implementation is not only realistic and feasible; it is, we feel, a professional necessity whose time is overdue.”

THE WELL-BEING PROJECTA LANDMARK STUDY

1989Funded by the California Department of

Mental Health Office of PreventionFirst consumer research project:

Conducted by the California Network of Mental Health Clients Jean Campbell, Principal Investigator Ron Schraiber, Co-Investigator

Discovering the Consumer Perspective Asking New Questions

negative outcomes identified self-management skills revealed importance of personhood established effects of prejudice and discrimination

quantified incongruity of values, perspective, and

identified needs between consumers, family members, and mental health professionals found

Findings 40% of mental health clients surveyed felt that “all”

or “most of the time” people treated them differently when they found out they have received mental health services:like they are violent (16%)like a child (21%)like they don’t know what is in their own best interest (31%)like they are incapable of caring for children (20%)like they are incapable of holding a job (33%)

Findings

More than half of the clients surveyed indicated that they “always” or “most of the time” recognize signs or symptoms of psychiatric problems and that they can take care of these problems before they become severe.

Findings

48% indicated that they have avoided treatment due to fear of involuntary commitment.

30% reported that they had “little” or “no” control over the kind of mental health services they receive.

HOW CONSUMERS BECAME PART OF THE MHSIP COMMUNITY

MHSIP = Mental Health Statistics Improvement Program

First consideration of adding consumers & family members came in the early 90’s.

On the local and state levels consumers were beginning to be involved in policy and evaluation

FIRST STEPS

STATE PLANNING COUNCILS INCLUDE CONSUMERS

CONSUMER REPRESENTATIVES JOIN MENTAL HEALTH BOARDS

ANN LODER PRESENTS PLENARY “HOPE WITH A CAPITAL H” AT NATIONAL MENTAL HEALTH STATISTICS CONFERENCE (1991)

JEANNE DUMONT BECOMES FIRST CONSUMER MEMBER OF MHSIP AD HOC ADVISORY GROUP (1992)

MOVING FORWARD

OFFICES OF CONSUMER AFFAIRS ARE ESTABLISHED ACROSS THE NATION

THE CONSUMER/SURVIVOR MENTAL HEALTH RESEARCH AND POLICY WORKGROUP IS FORMED (1991)

THE MHSIP MENTAL HEALTH CONSUMER-ORIENTED REPORT CARD (1994-PRESENT) CONSUMERS AND MENTAL HEALTH PROFESSIONALS COLLABORATE TO PRODUCE AN OUTCOME TOOL GROUNDED IN CONSUMER VALUES

CONSUMER OPERATED SERVICES PROGRAM (COSP) MULTI-SITE

RESEARCH INTITIATIVEhttp://www.cstprogram.org/cosp/

Consumer and non-consumer interviewers were employed from 1998-2002 for this SAMHSA project. The study took place in 7 different states. (Connecticut, Florida, Tennessee, Missouri, Illinois, Maine, Pennsylvania)

Among 6 of 7 study sites providing information about consumer hiring, the composition of their interviewing staff, it was reported that 2/3 of the project interviewers were consumers.

CURRENT INFORMATION ON CONSUMER-LED EVALUATION

MHSIP - http://www.mhsip.org/index.aspNational Empowerment Center -

http://www.power2u.org/SAMHSA - http://www.samhsa.gov/CONTAC - http://www.contac.org/Directory of Consumer- Driven Services –http://www.cdsdirectory.org/Self-Help Clearinghouse -

http://www.mhselfhelp.org/index.htmCenter for Psychiatric Rehabilitation Boston U. -

http://www.bu.edu/cpr/index.shtml

New York Association of Psychiatric Rehabilitation Services (NYAPRS)

PEER BRIDGER PROJECTPeer Bridger Project Outcomes

A participatory evaluation of the project prepared by Cheryl MacNeil, Ph.D. identified and examined several

areas where the project benefited those involved:

The development of uniquely different relationships than one typically experiences in a psychiatric hospital

The temporary relief from a state of social isolation and physical enclosure often experienced by those hospitalized

The ability to share with each other the experiential wisdom and survival skills necessary for the process of recovery

Rehospitalization Rates:

"The most substantial finding is that the follow-up rehospitalization rate of Matches while enrolled in the Peer Bridger Project was

significantly less than the baseline hospitalization rate (i.e.. the 2-year period prior to enrollment).

That is, during the 2-year baseline period, the Matches were hospitalized an average of 60% of the time, while enrolled in the

program, however, they were rehospitalized only 19% of the time. That's an improvement of 41%!". (National Health Data Systems,

December 1998)

For further information about the Peer Bridger Project please fill out our online request form or contact Tanya Stevens,

Peer Bridger Project Director, by phone at 518-436-0008 x18 or by e-mail at TanyaS@nyaprs.org.

EDITORIAL

“I believe the real mental health transformation will come when our practice is guided by people first values. Values fundamental to our culture, such as self determination, freedom, opportunity and choice, must be fundamental to our practice. “

“When I look at the field of mental health services research, I see a field in which significant research typically follows significant changes in practice.”

EDITORIAL CONTINUED“From my perspective, what currently is contributing

most to delaying the creation of recovery-oriented, people first practices is not the absence of critical science underlying our practices, but the absence of the critical values underlying our practices. And if our practices were grounded in these people first values, subsequent research could study how to best incorporate these values into our program and system policies, procedures, and documentation. Science can certainly help close the gap between values and practice.”

William A. Anthony Summer 2005 Psychiatric Rehabilitation Journal

Group discussionEvaluation questions What does your organization have to

offer to improved health and wellness of clients or the community?

What would you like to evaluate with respect to those impacts?

What else could your organization learn in order to improve its effectiveness?

What evaluation questions would be related to the above questions?

Group discussionEvaluation methods How might you gather data on those

evaluation questions? What would you need to gather, analyze,

report, and use the data? What kind of support would you need?

Examples: A list of existing instruments, how to conduct a focus group, how to create a measure, how to analyze data, how to do a report

Washington State Consumer and Family Evaluation Grant ProgramHow best to provide support to consumer

organizations and consumer evaluators? Ideas:

Conferences and educational opportunitiesAn RFP and Grant process for consumer and

family organization evaluation and consumer-led evaluations

Consultation and technical assistance to support consumer and family-led evaluation projects