Program Evaluation (by M. Secret)

What is the purpose of program evaluation

To investigate social programs

To assess effectiveness of social policies and programs.

The language of evaluation: Fill in the

Blank the impact of the program process on the cases processed

Outcomes

the services delivered or new products produced by the program processOutputs

Population for whom the program is designed.Target Population

information about service delivery system outputs, outcomes, or operations that is available to any program inputs

Feedback

individuals and groups who have some basis of concern with the program, often setting the research agenda and controlling research findings

Stakeholders

the complete treatment or service delivered by the programProgram process

resources, raw materials, clients, and staff that go into a programIn puts

Question FIRST!!!

Questions to be answered• Is the program needed?

Do a needs assessment

• How does the program operate?Do a formative evaluation

• What is the program’s impact?Do a summative evaluation

• How efficient is the program?Do a cost benefit or a cost effectiveness analysis

The specific methods depends on the question of interest

Definition of Needs Assessment Terms*

Needs Assessment:Systematically researching questions about the needs of a target

population for program planning purposes

Key Informants: expert opinions from individuals who have special knowledge about

the needs and about the existing services

Rates under treatment: secondary analysis of existing statistics to estimate need for services

based on number and characteristics of clients who are already being served.

Social Indicators: existing statistics that reflect conditions of an entire population.. i.e.

census data, Kids Count data. Rubin and Babbe, (2007) Essential Research Methods for Social Work. Brooks Cole:CA

Is the program needed?Do a needs assessment

Possible approachesKey informant interviews – use individual interviews

in-person, or by phone, structured or unstructured. Community forum – use a focus groupRates under treatment – use secondary data from existing

agencies (who uses existing services)Social indicators – use secondary data from census or other well-

being indicators (infant mortality rates, suicide rates) Surveys of community or target groups – consideration has to be

given to sampling unit..

Who ‘speaks’ for the group

• WHO will you consider ‘key informant’• Client, service provider, academic

• Who will you survey??• What statistics are important.. • How and who defines the problem • What do you mean by community??

Political and value issues are a major consideration

–a practice/political issue, not necessarily a research issue

Class Exercise: Social Worker at a local YMCA doing a needs assessment re “Do fathers in your community need support/services to help them be better parents?

1. Who will you interview for key informant interviews? What will you ask them? (what variables will you measure)

2. Who would you invite to a community forum/focus group.. What questions would you ask?

3. What ‘rates under treatment’ are important -- what agency organizations would you go to for information? What information -- variables do you identify?

4. What Social indicators are important.. Where would you go to collect them–

5. Who would be your sample for a community survey.. What would you ask.

Why Focus On Parenting Programs for Incarcerated Fathers?

• 93% of incarcerated parents are malea

• 75% of male inmates are fathersa

• Very few male institutions have any type of family programming available to inmates and their families

• Many female institutions currently have parenting programs in place

a b U.S. Bureau of Justice Statistics. 2000. Incarcerated Parents and Their Children. Office of Justice Programs, Special Report.

ANSWERS THE NEEDS ASSESSMENT QUESTIONSWITH

•SOCIAL INDICATORS•RATES UNDER TREATMENT

Moving on to existing program

AS you begin this formative or summative evaluation of an existing program, you must

• Identify the logic model

• Consider the‘evaluability’ of the program

What is the Logic Model?A schematic representation of the various

components that make up a social service program.

• Logic models may describe• theory and its link to change (theory approach

model) where attention is on the “how and why” a program works

• outcomes (outcome approach model) where the focus of the logic model is to connect resources and activities to expected changes

• activities (activities approach model) or describing what the program actually does

Resources Program Procedures – Interventions Delivered by the program staff or volunteers

Psychological and Social Process undergone by the incarcerated fathers

Interim Outcomes -- Program Goals

Long-Term Outcomes

Direct service time (paid and volunteer) Managerial and administrative support time Equipment (VCR, monitors, overhead, tape recorders) Supplies (Handouts, newsprint, markers, video tapes, audio tapes, workbooks, postage, newsletters, children’s books, etc)

12-week parent education classes, with specified goals and objectives for each class, delivered by paid and volunteer professionals, that provide content, materials, experiential learning activities on pre-specified topics relating to the increase of protective factors /decrease of risk factors re child abuse, and interactive discussions about how the fathers were parented as children and philosophies that emphasize the importance of fathers in the lives of children.

Understanding children’s behavior; Identification of situations which trigger anger and stress; Knowledge of child development and discipline; Support and insight from other fathers about parenting techniques; Increased parenting communication skills; Awareness of impact of family conflict on parent-child relationships; Awareness of feelings of sadness, anger, self-esteem; Awareness of effects of parental substance abuse on children; Awareness of how parenting skills are passed down from one generation to the next; Exposure to “Long Distance Dads” workbooks.

Decrease Child Abuse Potential

Increased knowledge and use of parenting skill

Decrease in feelings of isolation

Increased empathy with children

Increased recognition of importance of role of fathers

Increase understanding of how life experiences effect parenting

Increased recognition of correct parenting

Decreased Recidivism Decreased Child Abuse

Logic model- outcomes exampleProgram inputs

Program processes

WHAT’S MISSING IN THIS MODEL

Identifying the causal

mechanism

Short term outcomes.Measured

by the research

Long term

outcomes/ difficult

to measure

Program OutputsNumber of fathers who participate in program

What’s an ‘Evaluabilty’ Assessment

a systematic process that helps identify whether program evaluation is justified, feasible, and likely to provide useful information*.

• determines whether a program is ready for evaluation—either a process or outcome evaluation, or both.

• Is the program able to produce the information required for a process evaluation,.. AT WHAT STAGE OF IMPLEMTATION IS THE PROGRAM?

• Can a program meet the other criteria for beginning an outcome evaluation. • determines whether a program has the basic foundation for an evaluation to

take place

* Evaluability Assessment: Examining the Readiness of a Program for Evaluation. Juvenile Justice Evaluation CenterJustice Research and Statistics Association. Program Evaluation Briefing Series #6. May, 2003, p. 6http://www.jrsa.org/pubs/juv-justice/evaluability-assessment.pdf

Newly emergent programs that are not fully operational are not ready for,

and indeed can be tarnished by a summative evaluation geared to assessing program outcomes.

HOW SO??

FACT PROGRAM BACKGROUND• Fathers And Children Together (F.A.C.T.) is prison-based parenting program that

helps incarcerated fathers learn and enhance parenting skills and bond with their children. Begun in 1992, it is a joint project of Prevent Child Abuse Kentucky (PCAK) and Blackburn Correctional Complex (BCC), a minimum security prison within the Kentucky Department of Corrections. F.A.C.T. provides prison-based parenting education classes, special visits for fathers and their children as well as unique family outreach and leadership opportunities. Classroom sessions (2 hrs) operate on a 12-week schedule covering topics such as discipline, family relationships, communication, and parental substance abuse effects on children. Special visits are offered to F.A.C.T. participants every 6-8 weeks that allow fathers and their children to interact and play together on prison grounds without many of the constraints typical of regular prison visitation. The Parent Leadership Team promotes peer leadership among a small group incarcerated fathers and supports the activities of the classroom sessions and the parent-child visits. The essential philosophy of F.A.C.T. is to reduce the potential for child abuse and neglect and promote father involvement in the lives of their children by creating positive father and child experiences and opportunities for learning inside prison. This model program has provided a baseline for replication in other state correctional institutions.

Evaluability of F.A.C.T. program

ESTABLISHED PROGRAM had measurable outcomesdefined service componentsan established recruiting, enrollment, and participation

process; good understanding of the characteristics of the target

population, program participants and program environment; ability to collect and maintain information;adequate program size

RESEARCH SAVVY SERVICE DELIVERY STAFF hadproblem solving values and skillsprior experience with evaluation confidence in programcommitment to ‘new knowledge’openness to change

What is a Process Evaluation (implementation assessment):

evaluation research that investigates the process of service delivery

• Important for complex programs• Can identify the specific aspects of a program that have

an impact• Quantitative data is useful but qualitative data are used to

elucidate and understand internal program dynamics, unanticipated outcomes, and interaction among program participants

What is an Outcome Evaluation: Also known as impact evaluation and summative evaluation

evaluation research that examines the effectiveness of the treatment or other service

• Experimental design is preferred method for maximizing internal validity

• Program is independent variable (treatment)• Outcomes are dependent variables• Random assignment allows evaluation of programs impact

independent of the types of people in different groups• Selection bias in non-experimental designs is a common

problem because people self-select to participate in programs

RESOURCES

Evaluation Team Observation Notes

Participant Sign-In Sheet (every class)

PARENT-CHILD

VISITS

Cost Analysis Data Form (every 6 mths

PARENT EDUCATION CLASSES

Self-Administered Outcome Instrument (first class, last class, 6 month follow-up)

Demographics Family and Criminal History Adult-Adolescent Parenting Inventory (AAPI-2) Child Abuse Potential Inventory (CAPI) Childhood Trauma Questionnaire (CTQ) Parenting Awareness ‘Behavior Change’

Protocol

Participant Sign-In Sheet (every class)

Self-Administered Class Session Report (CSR) (end of every

Video-taping (2 classes per series)

Staff Reflections video tape (end of series)

Measures for outcome and causal

mechanisms Does the program

cause change? How does change happen

Measures for

Process/Implementation Evaluation

What services are being delivered,

by who, how?

Measures for input

And program efficiency

USING MULTIPLE MEASURES and SEVERAL DATA COLLECTION STRATEGIES TO EVAULATE THE FACT PROGRAM

Formative/Implementation Evaluation

How do you know whether or not the service was delivered in the manner intended… i.e. according to protocol

Must measure the Independent Variable –

Issues of Treatment Fidelity a major concern

Participant Sign-In Sheet (every class)

Self-Administered Class Session Report (CSR) (end of every

Video-taping (2 classes per series)

Staff Reflections video tape (end of series)

CLASS SESSION REPORT CLASS SESSION TWO

(CIRCLE THE WORDS IN ONE NUMBERED BOX FOR EACH STATEMENT)

AFTER THIS CLASS I UNDERSTAND ….

1 2 3 4

WHY MY KIDS DO WHAT THEY DO AT DIFFERENT

AGES

MUCH

BETTER

SOME

BETTER

NO

BETTER

WAS NOT TALKED

ABOUT IN CLASS

THE BASIC NEEDS OF MY CHILDREN MUCH

BETTER

SOME

BETTER

NO

BETTER

WAS NOT TALKED

ABOUT IN CLASS

WHY MY PARENTING IS IMPORTANT TO MY CHILDREN MUCH

BETTER

SOME

BETTER

NO

BETTER

WAS NOT TALKED

ABOUT IN CLASS

THE DIFFERNECE BETWEEN “SELF-CONCEPT”

AND “SELF-ESTEEM”

MUCH

BETTER

SOME

BETTER

NO

BETTER

WAS NOT TALKED

ABOUT IN CLASS

1. What was the one most important thing you learned in today’s session? ____________________________________________________________________________

____________________________________________________________________________

____________________________________________________________________________

2. Tell how this information will help you as a father?

____________________________________________________________________________

____________________________________________________________________________

____________________________________________________________________________

____________________________________________________________________________

3. What feelings do you have about yourself as a parent after today’s session?

____________________________________________________________________________

_____________________________________________________________________________

_____________________________________________________________________________

___________________________________________________________________________

Self-Administered Class Session Report (CSR) (completed by participants at end of every class session

One Measure forFormative/Implementation Evaluation

Goals of the curriculum for this particular class

Helps assess degree to which curriculum was being delivered

Importance of Treatment Fidelity Analysis

• Can minimize incorrect research conclusions• Can help to explain why a program works• Helps to identify programmatic changes• Underscores potential feasibility for replication

Measure Instrument/Data Source Activity

Adherence to program a. CSR 

b. Video Taped Sessions

Measure frequency of Item #4 selected (not covered) Review for consistency with written curriculum

 Measuring the amount of the program delivered (dosage) 

a. PPI Measure dosage variable for graduates

 Ascertaining quality of delivery 

a. CSR b. Video Taped Sessions

Measure mean scores for classes attended.  Review for quality of delivery

 Measuring participant responsiveness 

a. CSR

b. Video Taped Sessions

Measure for written comments vs. no comments.  Review for participant responsiveness

 Measuring program differentiation 

a. “Reflections” transcriptsReview for substantial differentiation between series.

Treatment Fidelity Assessment Treatment Fidelity Assessment PlanPlan

MEASURING THE MEASURING THE INDEPENDENT VARIABLE INDEPENDENT VARIABLE

Note: Fidelity is measured only on parent education classes.

Important aspects of treatment fidelity

Measuring program resources Name ____________________________ Date ___________________

Cost Analysis Data Form

Cost is defined as the value of each resource that is consumed when the program implements a service procedure.

Component = Classroom Session, Storybook Activity, PLT activity or Parent/Child Saturday VisitationResource = Your Time, Volunteer time, Space Used, Equipment Used, Supplies Used, etc.The Conversion and Total Cost columns will be computed at UK so you only need to complete the first three columns.

Amount TotalFACT Component Resource Used Conversion Cost

Cost Analysis Data Form (every 6 mths

Component Cost per unit

Units per Series

Cost per series

Parent Education Classes

$ 311

12 classes

$ 3,732

Parent/child Visits

$ 1,169 (869)

2 visits $ 2,338 (1,738)

Parent Leadership Team

$ 57

4 meetings $ 228

TOTAL COSTS

$ 1,537 (1237)

$6,269 (5,698)

Summative Evaluation

• Experimental design is preferred method for maximizing internal validity

• Program is independent variable (treatment)• Outcomes are dependent variables

FACT PROGRAM GOALS TRANSLATED INTO TESTABLE HYPOTHESES

H:1. Fathers who participate in at least 10 classes of the parent education classes will increase their knowledge of effective parenting skills and decrease their risk for child abusing behavior.

H:2. Fathers who participate in at least 10 classes of the parent education classes will report less feelings of isolation.

H:3. etc etc etc

Evaluation Design SnapshotUse of a recurrent institutional cycle design first identified by (Campbell &

Stanley, 1963) and further articulated by Shadish, Cook, & Campbell, 2002 p. 152 . This design measures the effects of the program assessing pre-and post test measures of different cohorts (series 1, series 2, etc thru series 8) as they go through the FACT program.

Series 1, Spring, 04 Series 2, Summer, 04 Series 3, Fall, 04

NR O1pr X O1ps

………………………………………………………………………..….………....NR O2pr X O2ps

……………………………………………………………………………………NR O3pr X O3ps

“A program main effect is suggested by a certain pattern of results if O1ps, and O2ps, and O 3ps are higher than O1ps, O2ps, and O3ps AND if O1ps is higher than O2pr, and O2ps is higher than O3pr

NR = not randomly assigned0 = data collected by survey instrumentX = 12 week parent education series

Program Effects* in goal-related knowledge, attitudes, skills of fathers who complete FACT program:

Looking at PATTERNS OF CHANGE

50.0

70.0

90.0

110.0

130.0

150.0

170.0

190.0

210.0

230.0

250.0

pre-test post-test

CAPI ABUSE***

series 1

series 2

series 3

series 4

series 5

Series 6

Series 7

Series 8

Series 9

TOTAL

H1: Stat Sig decrease in the potential for physical child abuse** from the

beginning of class sessions to the end of the at the program

*per self-reports of fathers:

**Potential for child abuse based on feelings of distress, rigidity, unhappiness, reports of relationships with children and other family member comparable to known child abusers

All but one line is going down

AVERAGEPre-test score on the CAPI

Abuse instrument

for each series

AVERAGEPost-test score on the CAPI

Abuse instrument for

each series

A Closer Look

HIGH RISK SUB-SAMPLE (n =67) COMPARED TO NOT AT RISK SUB-SAMPLE (n = 73)

HI-RISK =fathers who scored Above the clinical

cut-point for child abuse

potential

Clinical cut point = 215

On average, fathers who scored above

the clinical cut-point BEFORE the classes,

scored below AFTER the classes

Scores of other fathers in the group DID NOT CHANGE

The Cost of Program Effects

• Calculate program costs in relation to program effects as a per father cost. • Of the 140 fathers who completed the program,

• 90 fathers reported program effect, i.e. decrease in CAPI scores from pre-test to post-test over the course of the evaluation period.

• Given that we had 9 series of classes, we estimate that 10 fathers per series benefited from the program.

• It costs approximately $570 to decrease the child abuse potential of one father in the F.A.C.T. program.

• Fathers who do not participate in the visits obviously use less program resources ($373).

• The cost increases to $949 per father if we consider only those fathers (n = 52) who began the program ‘at-risk’ for child abusing behaviors, estimated to be roughly 6 fathers per series.

• We had no way to measure the ‘start-up’ costs of a program similar to F.A.C.T. but anticipate that the costs of new F.A.C.T.- like programs would be greater than the costs we calculated for our established F.A.C.T. program.

Black Box or Program Theory?

• Black box: • if program results are of primary importance, how it works may be of

secondary importance

• Program theory: • a descriptive or prescriptive model of how a program operates and produces

effects (theory-driven)• specifies how a program is expected to operate• identifies which program elements are operational• specifies how a program is to produce effects• improves understanding of the relationship between the independent variable (the

program) and the dependent variable (outcomes)

Example: F.A.C.T.

.00

Vulnerability

.00

Father Role Model.35

Father Scaffolding

.00

Father Empathy

.40

Father Capacity

E1

E2

E3

E4

E5

.63

.60

Experiences of Fathers and Knowledge and Attitudes

.02

-.03

.06

-.01

From questions

about history of child abuse

Type of parent

capacity At

beginning of the

program

Relationship between fathers child abuse growing up and father capacity and father empathy

Relationship between having good role models growing up and more education/background as adults

.00

Vulnerability

.00

Father Role Model

.36

Father Scaffolding

.40

Father Capacity

E1

E2

E3

E5

.60

-.10

.54

Cognitive learning

.61

Emotional learning

E6

E7

.73

.14

-.15

.63

.76

Father Experiences, Knowledge, Attitudes and Learning

Example: F.A.C.T. Program services/proceduresDifferent aspects of

the curriculum

Father’s history and level of parenting capacity at Beginning of program is related to what aspects of the program they benefited from

.00

Vulnerability

.00

Father Role Model .36Father Scaffolding

.40

Father Capacity

E1

E2

E3

E5

.60

-.10

.54CognitiveLearning

.61

EmotionalLearning

E6

E7

.73

.15

-.16

.63

.76

FACT Logic Model at Four Levels

.20

DecreasedAbuse Potential

.10

Knowledgeof Fathering

E4

E8

-.31

-.45

Example: F.A.C.T. Program outcomes

Results differ based on particular inputs and program

processes

What goes into the program

What happens in

the program What goes into the program

Another closer look Effects

remained after fathers left program

Caveat – fathers only left the program/prison..

Transferred to other prisons but remained in controlled setting.. Don’t know what happens when they return to the community

Researcher or Stakeholder OrientationConsider the pros and cons of each of the approaches

Stakeholder approaches encourage researchers to be responsive to program stakeholdersutilization-focused evaluationaction research (participatory research)appreciative inquiry

Social science approaches emphasize the importance of researcher expertise and maintenance of some autonomy in order to develop the most trustworthy, unbiased program evaluation. goal free evaluation

Integrative approaches attempt to cover issues of concern to both stakeholders and evaluators, and to include stakeholders in the group from which guidance is routinely sought