evaluating community-based initiatives may 3, 2005 12:00 noon cst holly ruch-ross, scd david keller,...

Evaluating Community-based Initiatives

May 3, 2005 12:00 noon CSTMay 3, 2005 12:00 noon CST

Holly Ruch-Ross, ScDHolly Ruch-Ross, ScD

David Keller, MD FAAPDavid Keller, MD FAAP

Thomas Young, MD FAAPThomas Young, MD FAAP

May 3, 2005 American Academy of Pediatrics

Declaration of Vested Interest Holly Ruch-Ross has no interests Holly Ruch-Ross has no interests

to declareto declare David Keller has no interests to David Keller has no interests to

declaredeclare Thomas Young has no interests to Thomas Young has no interests to

declaredeclare

Evaluation Overview

Holly Ruch-Ross, ScDHolly Ruch-Ross, ScD


Presentation Objectives

At the completion of the presentation, the At the completion of the presentation, the listener will be able to:listener will be able to:

1.1. Identify the distinguishing Identify the distinguishing characteristics of evaluation research characteristics of evaluation research and the benefits of conducting and the benefits of conducting evaluation.evaluation.

2.2. Write clear objectives.Write clear objectives.

3.3. Identify significant evaluation challenges Identify significant evaluation challenges and strategies to address them.and strategies to address them.


Defining Evaluation

ResearchResearch: to conduct a careful, patient, : to conduct a careful, patient, systematic, diligent inquiry or systematic, diligent inquiry or examination in some field of knowledge, examination in some field of knowledge, to establish facts or principles; to to establish facts or principles; to laboriously or continuously search after laboriously or continuously search after truth.truth.

Evaluate: Evaluate: to determine the worth of; to to determine the worth of; to find the amount or value of; to appraise.find the amount or value of; to appraise.


Defining Evaluation

Evaluation is Evaluation is action action research, intended research, intended to provide information that is to provide information that is usefuluseful for:for: Program development and Program development and

improvementimprovement Program replication Program replication Resource allocationResource allocation Policy decisions.Policy decisions.


Reasons to Evaluation Checking Your Process:Checking Your Process: Are you doing Are you doing

what you said you would do?what you said you would do? Determining Your Impact:Determining Your Impact: Are you having Are you having

the desired effect in the target population?the desired effect in the target population? Creating a Fan Base:Creating a Fan Base: Can you generate Can you generate

information and evidence to share with information and evidence to share with funders and other stakeholders?funders and other stakeholders?

Replication Justification:Replication Justification: Is there evidence Is there evidence to support replication of this program?to support replication of this program?

Effective Documentation:Effective Documentation: Can you collect Can you collect information to support your program and information to support your program and meet funder or other requirements?meet funder or other requirements?


The Evaluation Cycle

STARTSTART

Implement program and begin to collect evaluative data

Review data. Are you doing what you planned? Having the

intended effect?

Adjust program; Refine evaluation

Plan program and evaluation


Types of Evaluation

ProcessProcess

Is the program Is the program being being implemented the implemented the way it was way it was designed?designed?

OutcomeOutcome

Is the program Is the program having the having the intended effect?intended effect?


Process Evaluation Information Needs Describe the program and Describe the program and

implementation, who participates in implementation, who participates in the program, what services are the program, what services are received.received.

Information such as number served, Information such as number served, patient characteristics, number of patient characteristics, number of contacts with a program, number of contacts with a program, number of trainings, number of referrals.trainings, number of referrals.


Outcome Evaluation Information Needs Detect whether the intervention Detect whether the intervention

made a difference, what changes made a difference, what changes can be measured (knowledge, can be measured (knowledge, attitude, behavior, health status, attitude, behavior, health status, incidence, prevalence)incidence, prevalence)

Longer term outcomes may need Longer term outcomes may need to be assessed using shorter term to be assessed using shorter term indicators. indicators.


Outcomes and Indicators: Examples

OutcomeOutcome IndicatorIndicatorPrevent pregnancyPrevent pregnancy Contraceptive Contraceptive

compliancecompliance

High school graduationHigh school graduation School attendance, GPASchool attendance, GPA

Reduce disease Reduce disease incidenceincidence

Immunization rateImmunization rate

Reduce clinical Reduce clinical depressiondepression

CES-D scoreCES-D score

Prevent dental cariesPrevent dental caries Plaque indexPlaque index


Goals and Objectives

Goal: broad statement of what the Goal: broad statement of what the program would like to accomplish program would like to accomplish for a specific target population.for a specific target population.

Objective: measurable step Objective: measurable step toward the achievement of a goal.toward the achievement of a goal. WhoWho will do will do WhatWhat by by WhenWhen


A GOOD OBJECTIVE IS SMART: SSpecificpecific MMeasurableeasurable AAchievablechievable RRealistic for the programealistic for the program TTime specificime specific


Evaluation Challenges: Finding the Right Tools Tools need to measure the Tools need to measure the right construct. right construct. Ideally, tools should be pre-existing and well-Ideally, tools should be pre-existing and well-

established (valid, reliable and standardized).established (valid, reliable and standardized). Tools must be appropriate for the target Tools must be appropriate for the target

population (in terms of age, culture, population (in terms of age, culture, language, literacy, other issues).language, literacy, other issues).

Tools must be easy to administer in the Tools must be easy to administer in the setting.setting.

When existing tools are used, they must be When existing tools are used, they must be readily available, affordable, and supported readily available, affordable, and supported by the author.by the author.


Evaluation Challenges: Designing Your Own Tools Adapt an existing tool to be more Adapt an existing tool to be more

appropriate for target populationappropriate for target population Review literatureReview literature Talk to other granteesTalk to other grantees Talk to others with ideas about what Talk to others with ideas about what

you should ask about: experts, staff, you should ask about: experts, staff, recipients of servicesrecipients of services

Pilot test tools with representatives of Pilot test tools with representatives of the target population.the target population.


Evaluation Challenges: Using Qualitative Data Gain insight into feelings, attitudes, Gain insight into feelings, attitudes,

opinions and motivations.opinions and motivations. Study selected issues in depth and Study selected issues in depth and

detail.detail. Gather the broadest response possible Gather the broadest response possible

without predetermined categories.without predetermined categories. Gain rich information about a small Gain rich information about a small

number of people and cases.number of people and cases. Put a human face on the program.Put a human face on the program.


Most Common Qualitative Data Collection Strategies

In-depth interview: usually one-on-one In-depth interview: usually one-on-one structured conversation with a person structured conversation with a person who has important information about who has important information about the program.the program.

Focus group: professionally facilitated, Focus group: professionally facilitated, focused discussion among selected focused discussion among selected individuals about program issues. Not individuals about program issues. Not all discussion groups are focus groups, all discussion groups are focus groups, but they may still be useful.but they may still be useful.


Evaluation Challenges: Data Collection The sophistication of data collection should be The sophistication of data collection should be

appropriate for the scale of the project.appropriate for the scale of the project. Plan data collection up front, including who, Plan data collection up front, including who,

what and when.what and when. Have a system in place for tracking Have a system in place for tracking

participants (particularly if follow up is participants (particularly if follow up is planned).planned).

Identify the staff person responsible for data Identify the staff person responsible for data handling.handling.

Protect participant confidentiality.Protect participant confidentiality. Do not collect information that you will not Do not collect information that you will not

use.use.


Evaluation Challenges: Data Management and Analysis Budget for expenses associated with data Budget for expenses associated with data

entry and analysis.entry and analysis. Have a strategy for data management up Have a strategy for data management up

front. Begin data entry immediately.front. Begin data entry immediately. Remember that data analysis should Remember that data analysis should

follow directly from the questions you are follow directly from the questions you are trying to answer about your intervention.trying to answer about your intervention.

Know what comparative information may Know what comparative information may be available.be available.


Evaluation Challenges: Technology Involve technology as early as possible Involve technology as early as possible

in the planning process.in the planning process. Keep it as simple as possible.Keep it as simple as possible. Choose technology that will be around Choose technology that will be around

at least five years, and that is at least five years, and that is compatible with your current system.compatible with your current system.

Make certain that your choices match Make certain that your choices match the technical support that is available the technical support that is available to you.to you.


Evaluation Challenges: Getting Help From an Evaluator Specific evaluation training and Specific evaluation training and

applied research experienceapplied research experience Experience in a human service Experience in a human service

settingsetting Professional perspective and Professional perspective and

methodological orientation methodological orientation Interpersonal skillsInterpersonal skills Self interest (i.e., can he/she put Self interest (i.e., can he/she put

yours first?!)yours first?!)

CATCH Grant Evaluation Example

Thomas Young, MD FAAPThomas Young, MD FAAP


Presentation objectives

At the completion of the At the completion of the presentation, the listener will be presentation, the listener will be able to:able to:

Describe a CATCH ProjectDescribe a CATCH Project Describe project’s evaluation planDescribe project’s evaluation plan Describe project’s data collection Describe project’s data collection

and management techniquesand management techniques


Keys to Successful Evaluation Develop objectives that directly Develop objectives that directly

contribute to your goals for contribute to your goals for projectproject

Spend time and thought on Spend time and thought on objectives, evaluation will followobjectives, evaluation will follow

Objectives must beObjectives must be RelevantRelevant MeasurableMeasurable Feasible/doableFeasible/doable


CATCH Project Description

CATCH Implementation Grant 2003CATCH Implementation Grant 2003 Goal: To improve screening, Goal: To improve screening,

provider knowledge and access to provider knowledge and access to mental health care for children and mental health care for children and families in a primary care and families in a primary care and continuity clinic settingcontinuity clinic setting

This is broad goal for a small grantThis is broad goal for a small grant We had to develop objectives that We had to develop objectives that

were relevant, measurable, and were relevant, measurable, and doable.doable.


Developing CATCH Objectives Process Objective 1: 80% Process Objective 1: 80%

Implementation of Bright Futures Implementation of Bright Futures mental health screening: PEDS, mental health screening: PEDS, Ped Symptom Checklist, Ped Symptom Checklist, Edinburgh Postnatal Depression Edinburgh Postnatal Depression ScreenScreen

Evaluation Method: Chart audit Evaluation Method: Chart audit Findings: Chart audit documented Findings: Chart audit documented

successful implementation of successful implementation of screenings in 90% charts screenings in 90% charts


Developing CATCH Objectives Process Objective 2: To provide Process Objective 2: To provide

120 mental health consultant 120 mental health consultant visitsvisits

Evaluation Method: A data file Evaluation Method: A data file was developed and completed by was developed and completed by the Mental Health consultant to the Mental Health consultant to document the number and type of document the number and type of mental health encounters.mental health encounters.

Findings: 297 MH sessionsFindings: 297 MH sessions


Data File and Report


Developing CATCH Objectives Process Objective 3: To provide Process Objective 3: To provide

case management of mental health case management of mental health referralsreferrals

Evaluation Method: Data base Evaluation Method: Data base developed with Excel to track developed with Excel to track referrals, patient contactsreferrals, patient contacts

Findings: 83 children had outside Findings: 83 children had outside referrals and case management referrals and case management documented.documented.


Developing CATCH Objectives Outcome Objective 1: To improve Outcome Objective 1: To improve

provider confidence in managing provider confidence in managing behavioral problems in children.behavioral problems in children.

Evaluation Method: Development of a Evaluation Method: Development of a pre/post survey of the providers in pre/post survey of the providers in clinic on confidence in management of clinic on confidence in management of common behavior problems (no existing common behavior problems (no existing survey tool found)survey tool found)

Findings: Confidence increased from Findings: Confidence increased from 40% to 67% in 6 months. Adequate MH 40% to 67% in 6 months. Adequate MH resources improved from 54% to 95%.resources improved from 54% to 95%.


Developing CATCH Objectives

Outcome Objective 2: Identify Outcome Objective 2: Identify resources to sustain mental health resources to sustain mental health providerprovider

Evaluation: Was funding found to Evaluation: Was funding found to continue mental health projectcontinue mental health project

Findings: Primary Care Center Findings: Primary Care Center funding, grants, and partnership funding, grants, and partnership with community mental health with community mental health agency continued and expanded agency continued and expanded MH services.MH services.


Summary of CATCH Evaluation

Evaluations are different from scientific Evaluations are different from scientific researchresearch

Good objectives lead to good evaluationsGood objectives lead to good evaluations If possible find reliable tools to measure If possible find reliable tools to measure

outcomes, if not use creative common outcomes, if not use creative common sense.sense.

Accurately track data pre-grant to the endAccurately track data pre-grant to the end Keep it relevant but simple, with larger Keep it relevant but simple, with larger

follow-up grants hire an evaluator!follow-up grants hire an evaluator!

Healthy Tomorrows Grant Evaluation Example

David Keller, MD FAAPDavid Keller, MD FAAP


Presentation objectives

At the completion of the At the completion of the presentation, the listener will be presentation, the listener will be able to:able to:

Describe a Healthy Tomorrows Describe a Healthy Tomorrows ProjectProject

Describe project’s evaluation planDescribe project’s evaluation plan Describe project’s data collection Describe project’s data collection

and management techniquesand management techniques


Healthy Tomorrows Project Description:Family Advocates of Central Massachusetts A legal-medical collaborativeA legal-medical collaborative

Focused advocacy on legal issues likely to Focused advocacy on legal issues likely to affect child health outcomesaffect child health outcomes

Three objectives:Three objectives: Patient identification and referral from Patient identification and referral from

practicespractices Provider training on legal issues and how Provider training on legal issues and how

they affect healththey affect health Advice and counsel for patients and Advice and counsel for patients and

families in needfamilies in need


A medical-legal collaborativeLegal Assistance Corporation Legal Assistance Corporation

of Central Massachusettsof Central Massachusetts and five practices in and five practices in Central MassachusettsCentral Massachusetts

• WorcesterWorcester • Pediatric Primary CarePediatric Primary Care• Family Health CenterFamily Health Center

• WebsterWebster• South County PediatricsSouth County Pediatrics

• MilfordMilford• Milford PediatricsMilford Pediatrics

• FitchburgFitchburg• CHC Family Health CHC Family Health

CenterCenter


Focused Advocacy

• Housing stabilityHousing stability (e.g. lead poisoning, (e.g. lead poisoning, homelessness, mold and allergens)homelessness, mold and allergens)

• Financial securityFinancial security (e.g. disability benefits, (e.g. disability benefits, food stamps, Medicaid)food stamps, Medicaid)

• Dignity and safetyDignity and safety (e.g. immigration status, (e.g. immigration status, domestic violence)domestic violence)

• Access to servicesAccess to services (e.g. medical, dental, (e.g. medical, dental, mental health, special education services)mental health, special education services)


Model of Intervention

Medical Partner Legal Partner

Medical Legal Collaborative

Family

Health

Nutrition

Income

Housing

Health Care

Education Childcare

Disability

Child Protection

Family Law

Immigration

Child Health


Process Evaluation

Based on program objectives:Based on program objectives: Are the practices identifying and referring Are the practices identifying and referring

patients in need of legal services to the patients in need of legal services to the program?program?

Are we able to train medical providers on Are we able to train medical providers on legal issues and how they affect health?legal issues and how they affect health?

Have we provided advice and counsel for Have we provided advice and counsel for patients and families in need of legal patients and families in need of legal services?services?

Underlying question:Underlying question: What can we do better?What can we do better?


Process Objective 1: Are the practices identifying and referring patients in need of legal services to the program? Findings in Year 1: Findings in Year 1:

76 referrals76 referrals Came from all five Came from all five

sites.sites. Usually associated Usually associated

with trainings.with trainings. Office hours did not Office hours did not

generate referrals.generate referrals. Lessons learned:Lessons learned:

Time with providers Time with providers = more referrals.= more referrals.

Screening instrument Screening instrument needed.needed.

0

5

10

15

20

25

30

1stQuarter

2ndQuarter

3rdQuarter

4thQuarter

Observed Target


Process Objective 2: Are we able to train medical providers on legal issues and how they affect health? Year 1: 42 trainingsYear 1: 42 trainings

Webster: 9Webster: 9 Worcester FHC: 7Worcester FHC: 7 Worcester Peds Assoc: 15Worcester Peds Assoc: 15 Fitchburg: 6Fitchburg: 6 Milford: 5Milford: 5

FeedbackFeedback ““Code card is great”Code card is great” ““More time on cases”More time on cases”

Lessons learned:Lessons learned: Providers want practical Providers want practical

toolstools Providers want case-based Providers want case-based

trainingtraining

11 22 33 44 55

Values?Values? 00 00 00 77 99

Useful?Useful? 00 00 00 55 1212

MaterialsMaterials?? 00 00 11 22 1212

Trainer?Trainer? 00 00 00 55 1111

Satisfaction?


Process Objective 3: Have we provided advice and counsel for

patients and families in need of legal services? Findings: 58 cases closed Findings: 58 cases closed

for 50 clients. for 50 clients. Counsel and advice in 21 Counsel and advice in 21

cases. cases. Brief service in 26 cases.Brief service in 26 cases. Full representation in 10 Full representation in 10

cases.cases. Referred 1 clients to Referred 1 clients to

another agency.another agency. Lessons learned:Lessons learned:

Case mix more varied than Case mix more varied than expected based on needs expected based on needs assessment.assessment.

Less full representation Less full representation needed than we expected.needed than we expected.

0

5

10

15

20

25

30

35

HousingStability

Dignity/Safety

Observed Expected


Outcome Objectives: ???

Medical Partner Legal Partner

Medical Legal Collaborative

Family

Health

Nutrition

Income

Housing

Health Care

Education Childcare

Disability

Child Protection

Family Law

Immigration

Child Health

System Outcome

Health Outcome

Legal Outcome

Practice Outcome


Legal Outcome: Evaluation MethodsWas the case resolved in Was the case resolved in

favor of the client?favor of the client?

Is that client better off Is that client better off than before in terms than before in terms of:of:

• Housing stabilityHousing stability• Financial securityFinancial security • Dignity and safetyDignity and safety • Access to servicesAccess to services

From Nasdor, Moving Beyond Funder-Driven From Nasdor, Moving Beyond Funder-Driven Outcome Measures.Outcome Measures.

LACCM databaseLACCM database

LACCM outcome toolLACCM outcome tool


Legal Outcome: Problems

What is a case?What is a case?• Need to understand “jargon” of legal Need to understand “jargon” of legal

services.services.

What is an outcome?What is an outcome?• Need to integrate public health and legal Need to integrate public health and legal

datadata• Need an outcome for “brief intervention”Need an outcome for “brief intervention”

Validity of outcome measure?Validity of outcome measure?• Need to assess inter-observer reliabilityNeed to assess inter-observer reliability


Practice Outcome

How have the How have the attorneys changed attorneys changed their practice?their practice?

How have the How have the physicians changed physicians changed their practice?their practice?

Is there a more Is there a more collaborative collaborative relationship?relationship?


Practice Outcome: Problems Counting: Counting:

No ready made databaseNo ready made database Evaluation:Evaluation:

Utility of self-reported commentsUtility of self-reported comments Data entryData entry

Next steps: Next steps: Commitment to changeCommitment to change Focus groups on practice changeFocus groups on practice change


Health Outcome

What is health?What is health? Individual or Individual or

group?group? Compared to Compared to

what?what? Measured how?Measured how? What about the What about the

confounders?confounders?


System Outcome

Changes in policyChanges in policy Changes in the Changes in the

lawlaw Changes in Changes in

legislationlegislation Changes in the Changes in the

balance of powerbalance of power


Evaluation: Lessons Learned Build evaluation into the program Build evaluation into the program

designdesign Cross-disciplinary communication Cross-disciplinary communication

is essential to evaluating is essential to evaluating multidisciplinary collaborationsmultidisciplinary collaborations

Measure the easy stuff first, but Measure the easy stuff first, but don’t give up on the hard stuffdon’t give up on the hard stuff


Where to find more information… For an audio file of the call including For an audio file of the call including

question and answer period, go toquestion and answer period, go to http://www.aap.org/commpeds/resources/teleconferences.http://www.aap.org/commpeds/resources/teleconferences.htmhtm

To submit additional questions, please To submit additional questions, please email email [email protected]@aap.org. Your . Your question will be forwarded to one of the question will be forwarded to one of the teleconference speakers.teleconference speakers.

Please fill out the evaluation form and Please fill out the evaluation form and fax back to Anne Gramiak at 847-434-fax back to Anne Gramiak at 847-434-8000. A form must be completed to 8000. A form must be completed to receive continuing education credits.receive continuing education credits.


Program Staff Contact Information

CATCHCATCH Lisa Marth: Lisa Marth: lmarthlmarth@@aapaap.org.org Kathy Kocvara: Kathy Kocvara: kkocvarakkocvara@@aapaap.org.org Aleksandra Stolic: Aleksandra Stolic: astolicastolic@@aapaap.org.org

Healthy Tomorrows Healthy Tomorrows Nicole Miller: Nicole Miller: nmillernmiller@@aapaap.org.org Anne Gramiak: Anne Gramiak: agramiakagramiak@@aapaap.org.org

CPTICPTI Alanna Bailey: Alanna Bailey: [email protected]@aap.org Trisha Calabrese: Trisha Calabrese: [email protected]@aap.org

Medical HomeMedical Home Lauri Levin: [email protected] Levin: [email protected]

evaluating community-based initiatives may 3, 2005 12:00 noon cst holly ruch-ross, scd david keller,...

Documents

evaluation evaluation

evaluation slide

evaluation plan program

evaluation cycle

program development

scd slide

md faap slide

intended effect