using models evaluation planning - amazon web...

Using Logic Models for Evaluation Planning

Make sure to enter your access code and PIN in order to hear audio. We can’t unmute you unless you have

entered the audio PIN.

Webinar:

THIS WEB SEMINAR IS HOSTEDBY:

Capacity for Health at APIAHF, a Capacity Building Assistance Program, with offices in San Francisco and Washington D.C.

WHO WE ARE:

We are funded by The Centers for Disease Control and

Prevention to provide FREE trainings and one-on-one

technical assistance for community-based organizations

(CBOs) on:

(1) Organizational Infrastructure and Program Sustainability,

(2) Evidence‐Based Interventions (EBIs) and Public Health Strategies, and

(3) Monitoring and Evaluation.

HOUSEKEEPING

All participants are automatically muted by webinar administrators.

For any questions that arise during the presentation, please type them in using the Questions feature.

For discussions at the end of each section, please “raise” your hand. You will be called on and un-muted. A longer discussion will follow the presentation.

MSOffice1

MSOffice1 I would double check with the people on your agency to make sure this is compatible with the service you will be using. , 10/13/2010

USING LOGIC MODELS FOR PROGRAMPLANNING, EVALUATION, AND

COMMUNICATION

Lyn Paleo, MPA, DrPH

Employment:First 5 Contra Costa, Evaluation MgrEval Research & Training, ConsultantHarder+Co, Senior Rsh AssocSF AIDS Foundation, Education DirSign Language Interpreter

Education:UC Berkeley Sch of Public HealthHarvard’s Kennedy Sch of Govt

Antioch Univ, WestTeaching:

Core Eval Course, UCB SPH (10 yrs)Workshops, trainings, etc.

TOPICS OF THIS WEBINAR

Intros

Formats Logic Models

What a Logic Model is … and is not

Using the logic model for evaluation and presentation

Contributions and Questions

Our funder makes us.

We find it useful for program planning, evaluation, and communication.

WHY CREATE A LOGIC MODEL?

WHAT A LOGIC MODEL IS … AND IS NOT

First Quarter

LOGIC MODELS: THEY COME IN MANY FORMATS(SOME ARE BETTER THAN OTHERS)

Resources / inputs

Activities Outputs Outcomes Impact

Resources Activities / Strategies Outputs Outcomes Impact

Carlos Juras, Vice President, YLI 5% FTE

Mike Jones, 42.5% FTEYLI Director of Civic Engagement, Marin(day‐to‐day oversight)

Paul Rose, YLI 15% FTE Director of Marin Programs(Project management)

Linda Julias Senior Director of Research, YLI 12.5% FTE (Evaluation)

Janis Watts, Program Director of NWP (Technical assistance and support)

3 Student Coordinators with stipends

Strategy 1: Recruit, train, and retain HEY! teams at 4 high schools to participate in Wellness Policy implementation and monitoring (inc. SNACs and DWCs).

Recruit or build on existing teams of students (6 students + 1 adult).

Four teams of at least six students are recruited and retained.

Improved presence and quality of student participation in district and school deliberations and actions.

Students successfully participate in the implementation of District Wellness Policies.

HEY! staff provide extensive training to HEY! teams, including nutrition education and policy advocacy.

HEY! team members increase knowledge and skills. NWP staff provide TA, tools and linkage

to resources to support these efforts., including training on nutrition, social marketing, and access to F&V.

HEY! staff organize a student coordinating body with 2‐3 representatives of each HEY! team to meet quarterly.

Student coordinating body functions.

HEY! staff prepare HEY! members to participate in policy implementation and monitoring at the district level.

HEY! members are ready to participate in activities.

Components of the written plan are completed.

Strategy 2: Provide assistance and support to district and school site administrators to facilitate their efforts to implement Wellness Policies.

HEY! staff work with school administrators to prepare HEY! teams to assume lead roles in implementing, monitoring and evaluating District Wellness Policies in their schools.

School administrators are ready to work with HEY! team members.

Changes in attitudes and beliefs about student contributions by students. (?)

Logic Model: Marin Youth Leadership Institute

Youth and community service

Staff

Partners

Youth improve skills in planning, decision making, problem solving

Youth learn about their community

Youth demonstrateleadership skills

Youth are connected with and feel valued by their community

TimeYouth gain confidence in doing community work

Youth engage in additional community activities

Adults

INPUTS OUTPUTS OUTCOMES

Youth identify project to work on

Plan project

Carry out the project

Evaluate how they did

Youth ages

12-16

Youth successfullycomplete projects

Grant

Problem Statement Key Strategies Formative Outcomes (Process)

Intermediate Outcomes Long‐Term Outcomes

The community needs and problems our program addresses include:

In order to address our community needs, we propose to achieve the following strategies:

We expect that once completed or underway these strategies will produce the following evidence of success in program implementation:

We expect that if completed or ongoing these strategies will lead to the following changes in 1‐3 years:

We expect that if completed these strategies will lead to the following changes in 4‐6 years:

In the GV School District and our alternative schools, we have higher rates of underage drinking.

Normative behaviors, including adult expectations that youth drinking is part of normal experimentation, a natural part of growing up, fail to intervene in underage drinking and in some cases, encourage it.

Youth in GV believe that alcohol is less harmful than other substances, including cocaine, heroin, and methamphetamines.

Youth in GV have easy access to alcohol; the most common source is provision of alcohol by adults, siblings, and retail theft.

NEO New Generation Downtown:

Regular youth‐driven events in downtown areas in which there is currently widespread underage drinking.

Youth Outreach and Training:

Peer Advocates trained to conduct outreach to fellow adolescents who may be engaged in risky behavior with education and linkages to other STOP Act resources.

Overtime:

Friday night event offering an alternative to parties which are the site of abundant underage drinking.

Increase in after school and evening activities for youth grades 6‐12

Peer Advocates provide educational presentations to 200 youth per year on the harms of alcohol use

Peer Advocates refer 1000 youth to NEO events, Overtime, the SAP, and treatment

Overtime is offered 25 times per year

Increased participation in Peer Related Alternative activities by 50%

Increased participation of at‐risk youth at alternative events

Popularity of AOD‐Free Events increases

Increase in youth‐initiated AOD‐Free Events

Youth recognize the harms of alcohol use

Youth feel connected to community

Adults’ perceptions of youth changes

Change of cultural norms

Peer pressure reverses – youth pressure friend to NOT drink

Decrease in problem behaviors in middle school aged children

Age of onset of alcohol use is delayed

Decrease in underage binge drinking

Decrease in underage drinking

Decrease in percentage of youth engaging in other drug use

STOPActLogicModel– June2010

First Five Contra Costa Community Engagement: Impact Model

Support $

Staff

Com’ty Liaisons

Groups Form & Conduct Activities:

CENTRAL

EAST

WEST

Support members

Conduct Advocacy and

outreach to “spread First 5

message”

Organize Community

Fairs

Plan and organize Activities

classes

Participate in H&A B5 Plan

Members gain leadership

skills

Members gain advocacy skills

Classes offer fun, affordable

and safe activities

Fairs host resource tables

to inform parents of resources

Members influence local and state policy

regarding childhood obesity

Parents increase

awareness of importance of

healthy & Active living

Children join activity class (or) Children

put on wait list

Other orgs see the need for classes and

partner to fund new ones

Children are active in class

Parents get info on importance

of physical activity and to

incorporate nutrition into

diet

Children become

enthusiastic for sports

Reduced Childhood

Obesity

Lifelong activity

Parents support

children’s active

living and sports

More classes are offered

Recruit new members

LegendDocument, assess, or evaluate Rev. 08-10-08

LOGIC MODEL; SIMPLE FORM

Resources / inputs Activities Outputs Outcomes Impact

Certain resources are

needed to operate the

program

If you have access to

them, thenyou can use

them to accomplish

your planned activities

If you accomplish

your planned activities, then you hopefully deliver the amount of product or service that

you intended

If you accomplish

your planned activities to the extent

you intended, then your

participants will benefit in certain

ways

If these benefits to participants

are achieved, then certain changes in

organizations, communities,

or systems might be

expected to occur

Source: Kellogg Foundation, Logic Model

Development Guide, www.wkkf.org

LOGIC MODELS IN PROGRAM PLANNING

NeedsNeeds

ResourcesResources

Program DescriptionProgram

DescriptionLogic ModelLogic Model

EvaluationEvaluation

Program ServicesProgram Services

Presentation & CommunicationPresentation & Communication

Use to continue, modify, abolish, or replicate the

program

Use to continue, modify, abolish, or replicate the

program

Planning ------------------- Evaluation --------------------------------Use

FindingsFindings

WHAT IS A LOGIC MODEL?

• A logic model is a diagram of how the resources of the program lead to the desired impacts in the target population.

$ ∆

$ ∆

It shows the relationshipsbetween investments/resources, activities, and results.

It provides a common approach for integrating planning, implementation, evaluation and reporting.

WHAT A LOGIC MODEL IS NOT

1. It is not reality. Is a simple model that represents program intention.

2. It is not complete. It does not display many cultural, social, and environmental factors that influence process and outcomes outside the program

3. Does not prove causal attribution of the intervention to the change

4. It is not a Theory of Change. But it relies on a social theory or other theory of change.

5. Doesn’t address: Are we doing the right thing?

IMPACT AND LOGIC MODEL

Support $

Staff

Com’ty Liaison

s


CENTRAL

EAST

WEST

Support members

Conduct Advocacy

and outreach to “spread

First 5 message”

Organize Community

Fairs


classes


Members gain

leadership skills

Members gain

advocacy skills

Classes offer fun,

affordable and safe activities

Fairs host resource tables to inform

parents of resources



Parents increase



Children join activity class (or) Children put on wait

list

Other orgs see the need for classes

and partner to fund new

ones


Parents get info on

importance of physical

activity and to incorporate nutrition into

diet

Children become


Reduced Childhood

Obesity

Lifelong activity

Parents support children’s active

living and

sports


Recruit new members

Legend

Document,

assess, or

evaluate

Rev. 08-10-08

Problem to address Poverty and violence along with limited access to healthful foods and physical activities for 14,000 children and youth in the Valley Hi community.

Program Goals 1) Continue to create supports and opportunities for youth so that they can realize their full potential and successfully transition into adulthood, using youth‐lead coalition.2) Implement a community garden for youth and adults to connect through participation in the Coalition.

Collaborative Partners

Elk Grove Unified School District; People Reaching Out; Valley Hi Youth Adult Coalition, City of Sacramento; Hands On Sacramento; Parks and Recreation

Resources Activities Outputs Outcomes Impact

Tami Sisswa, EGUSD Liaison btwn EGUSD and Coalition 6% FTE

Erika Ergaos, People Reaching Out.Integrating youth and community components of the coalition, cultivate community partners and evaluation / reporting activities. 20% FTE + 75% FTE in‐kind

People Reaching Out Ex. Dir; Liaison with staff, EGUSD and Coalition10% FTE

New Position. Youth and Community Specialist at People Reaching OutPlanning and conducting on‐site program activities; recruit and train high school students75% FTE

Coalition Facilitate monthly VHYAC meetings and action teams.

Regular coalition meetings and action team activities.

Increased community outreach and sharing of resources.

Institutional / community practice and policy changes for improved nutrition and physical activity.

Advocacy Utilize the needs assessment to create a list of potential advocacy opportunities.

Roundtable discussion on potential advocacy opportunities.

Coalition engages in advocacy.

Youth Dev. Recruit additional youth leaders from Samuel Jackman and Valley High School.

Increased number of youth participating in Coalition.

Youth are engaged as leaders in the project. Adults support youth.

Youth Dev. Facilitate youth training retreat annually.

Number of youth who attend a retreat.

Youth have leadership skills and are prepared to be leaders and members of the VHYAC.

Youth Advocacy Train youth in advocacy information and strategies.

List of topics of trainings, number of youth attending.

Youth have advocacy skills and recognize opportunities to advocate for changes in their community.

Youth advocate for increased access to healthful food.

Youth Advocacy Youth develop advocacy and education projects.

Youth coalition members create a presentation for elementary school youth on the benefits to leading a healthy lifestyle.

Schools enable youth coalition members to make a presentation at school.

Elementary school youth become knowledgeable about the benefits of a healthy lifestyle and have access to healthful food.

Youth create a video highlighting healthy food preparation options for youth distributed in the community and potentially within the EGUSD.

Schools show the video.

Community residents and school youth become knowledgeable about healthful food preparation options.

Impact Model Logic Model

LOGIC MODELS ANDTHEORIES OF CHANGE

$ ∆

We anchor a logic model to a Theory of Change that provides rationale for how that set of activities can lead to that set of short-term outcomes, and how those outcomes can lead to long-term outcomes or impacts.

Theory of Change

A logic model displays the relationship betweenactivities and outcomes, but it does not guarantee that the relationship is meaningful, strong, or true.

A THEORY OF CHANGEPROVIDES RATIONALE

Improved nutritional health

Program provides cooking classes to teens

Learn and practice menu planning & cooking skills Teens eat

better

Gain knowledge of healthy foods and shopping tips

Receive nutritious free food for family

Teens cook for family weekly

It provides rationale for the arrows in a logic model.

It is the scientific or other basis for the “logic” in the logic model.

SOCIAL THEORIES OF BEHAVIOR CHANGE

• The Health Belief Model: perceived susceptibility, seriousness, benefits of taking action, barriers to action, cues to action

• Stages of Change: precontemplation, contemplation, preparation, maintenance, action

• Locus of Control: internal locus of control, powerful others locus of control (health is determined by others), chance locus of control (it's all fate or chance).

• Social Learning Theory: Response consequences (such as rewards or punishments) influence the likelihood that a person will perform a particular behavior again in a given situation. Humans can learn by observing others, in addition to learning by participating in an act personally. Individuals are most likely to model behavior observed by others they identify with.

http://hsc.usf.edu/~kmbrown/hlth_beh_models.htm

THE HEALTH BELIEF MODEL

• Perceived susceptibility • Perceived seriousness• Perceived benefits of taking action• Reduce barriers to action• Cues to action

Source: http://hsc.usf.edu/~kmbrown/hlth_beh_models.htm

IMPACT MODEL BASED ON A SOCIAL THEORYStaff distribute info cards in neighborhood about diabetes.

Senior residents become aware of their risk for diabetes (perceived susceptibility)

25 seniors are recruited for 3 workshops

Participants learn of the benefits of taking action

Seniors become aware of the seriousness of diabetes

Wkshop 2: Physician/nurse talk about how early dx and tx can reduce effects

Risk for diabetes is reduced

Wkshop 3: Nutritionists demonstrate easy to prepare, healthful meals; food tasting competition

Wkshop 1: Guest speakers with diabetes talk about their health problems

People learn how to prepare tasty, healthful meals (barriers reduced)

Staff send monthly newsletter with reminders, new recipes

Seniors’ nutrition is improved

Seniors prepare meals based on newsletter guidelines (cues to action)

ACTIVITIES OUTCOMES IMPACT

THE “LOGIC” OF LOGIC MODELS

WorkshopsIncreasedknowledge

IncreasedCondom use

Reduction In HIV

How strong is the Theory of Change supporting this model?

• strength of intervention (a pill is “stronger” than a visit)• intensity of intervention (frequency, duration)• suitability of intervention to ameliorate the problem

THE “LOGIC” OF LOGIC MODELS

WorkshopsIncreasedknowledge

Decreased personal

aggression

World Peace

How strong is the Theory of Change supporting this model?

Why do you say it is not strong--did you research it?

Of course not. Experienced program planners know a lot about what works and what doesn’t.

Impact Model, unforeseen paths

Local outreach workers trained, Work in comm.

Women cometo clinic

Staff gettraining

Compassionate, skilled staff

Women accept services

Neviprine, breastfeedsupport delivered

Healthierbabies

Fewer HIV+ babies

Increased condom use

Education about condoms

Fewer pregnancies

Vitaminsgiven

1

3

2

4

5

6

7

First Five Contra Costa Community Engagement: Impact Model

Support $

Staff

Com’ty Liaisons


CENTRAL

EAST

WEST

Support members

Conduct Advocacy and

outreach to “spread First 5

message”

Organize Community

Fairs


classes


Members gain leadership

skills

Members gain advocacy skills

Classes offer fun, affordable

and safe activities

Fairs host resource tables

to inform parents of resources



Parents increase



Children join activity class (or) Children

put on wait list

Other orgs see the need for classes and

partner to fund new ones


Parents get info on importance

of physical activity and to

incorporate nutrition into

diet

Children become


Reduced Childhood

Obesity

Lifelong activity

Parents support

children’s active

living and sports


Recruit new members

LegendDocument, assess, or evaluate Rev. 08-10-08

FORMATS FOR LOGIC MODELS

Second Quarter

NO ONE RIGHT WAY

• There is no one right way to format a logic model.

• There are (at least) 5 ways to format a logic model that makes it hard for others to understand.

EVERYTHING LEADS TO EVERYTHING

The “Everything leads to everything” format puts all activities in one column, all outputs in another, all outcomes in another, and so forth.

Activities Outputs Outcomes Outcomes ImpactSTAFF WILL PROVIDE PARTICIPANTS WITH:

Mentor/ mentee relationships

Workshops on personal development

Meetings with African American role models

Community Education Trainings

Opportunity for participation in Health Department meetings

Opportunities to present at workshops on HIV-related topics

12 recruited interns

8 mentors

4 workshops

12 meetings with role models

15 hours of community education trainings

Attendance at 25 department meetings

Participation in 10workshops on HIV-related topics

Participants increase their professional skills, including computer, and presentation skills

Participants increase their knowledge of the HIV service delivery system

Participants gain a deeper understanding of health-related career options

Participants increase their personal awareness and knowledge about health and HIV

Participants achieve a greater sense of self-efficacy, resiliency, and self-esteem

Participants will develop skills in biomonitoring and community education

Participants increase academic achievement and receive GED, finish high school, or apply for college

Participants increase leadership skills

Participants engage in healthier behaviors and reduce risk for HIV

Participants will achieve a greater hope for future success

Participants educate community members about HIV and other diseases

Increase in African American leadership in health-related fields

Reduction in health disparities among African American residents in Berkeley

EVERYTHING LEADS TO EVERYTHING

• What is difficult about this format:It is a bit “old school” Kellogg. Logic model format has evolved in the past several years.

This format does not well use a strength of logic and impact models: the ability to show which inputs or activities lead to which outcomes.

EVERYTHING LEADS EVERY WHICH WAY

The “Everything leads every which way” format is a jumble of boxes and arrows.

Especially arrows.

Community Health Worker Training Program: IMPACT MODEL

Participants achieve a greater

sense of self-efficacy,

resiliency, and self-esteem

STAFF WILL PROVIDE PARTICIPANTS WITH:

Mentor/ mentee relationships

Community Education Trainings

Participants increase their professional

skills, including computer,

professional writing, and presentation

skills

Workshops on personal development

Participants increase academic

achievement and receive GED, finish

high school, or apply for college

Meetings with African American role models

Participants increase their

personal awareness and

knowledge about health and

chronic disease

Participants engage in healthier

behaviors and reduce risk for hypertension,

CVD, and other chronic diseases

Increase in African American leadership in health-related fields

Opportunity for participation in Health Department meetings

Workshops on health related topics

Reduction in health

disparities among African

American residents in

Berkeley

Participants increase

leadership skills

Participants educate

community members

about chronic diseases

Participants gain a deeper

understanding of health-

related career options

Participants increase their

academic skills

Participants will achieve a

greater hope for future success

Participants will develop skills in biomonitoring and community education

PRIMARY ACTIVITIES PROCESS OUTCOMES

EVERYTHING LEADS EVERY WHICH WAY

• What is difficult about this format:

It would take a lot of dedication and concentration to slog through this format to fully understand what the program designer’s logic is.

It does differentiate which activities lead to which outcomes, but the paths are hard to follow.

THE MORE SAID THE BETTER

The “More said the better” format includes many, many activities and outcomes.

(The example shows only one of the three pages developed for this program. The program consists of one full-time staff member and several part time outreach workers.)

Children demonstrate

knowledge of how to use materials in backpacks

Program Name: School Readiness

Parents learn about methods of handling stress/ Parents learn about discipline &

developmental expectations

Teach literary skills/

modeling

Teach social emotional

skills

Parents connect reading with

lifelong successTeach

communication/ curiosity

Children’s social and emotional skills

improveParent Child

School Readiness Education

Programs provide accessible school

readiness information and

workshops:

Parents feel welcome in

schools/ Parents feel connected to

schools

Transitions to K-garten

Schools communicate

expectations to parents of entering

kindergartens

Schools allow these programs Schools

encourage kindergarten registration

Parents learn about kindergarten expectations

Children transition

more smoothly

School based activities that reach parents and children prior to kindergarten

Parents use school-created materials

(Backpacks) to work with children at

home

Schools reach out to children and families prior to

school registration

Kinder and preschool

teachers meet to plan

Backpacks and fairs

Schools initiate contact with early care providers to ease transitions/

share expectations

Raising a reader

Tigo

Summer PreKinder (parents & children together)

WCCUSD only

Understand & support kider expectations

Programs reinforce parents

positive behaviors

Parents learn how to work with their children on school

readiness skills

Parents sometimes learn from one another

Children gain book handling/ pre-reading skills

Children’s communication skills

improve

Kindergarten

readiness first day

successful

Parents increase confidence in their role as child’s first

teacher

Parents are involved in children’s education

Children ready for

school

WE FUND

SKILLS ATTITUDES

CHANGE IN KNOWLEDGE

ACTIVITIES PEOPLE FUNDED

VALUESBEHAVIORSCHANGE IN

SYSTEMSGOAL/

IMPACT

THE MORE SAID THE BETTER


It would take a lot of dedication and concentration to slog through this format to fully understand what the program designer’s logic is.

It includes many operational details and many outcomes that are not very important. Some outcomes could be effectively combined.

A FEW WORDS EXPLAIN EVERYTHING

The “Few words explain everything” format includes a few verbs and objects.

This format often incorporates more shapes and colors than concepts.

A FEW WORDS EXPLAIN EVERYTHING


More time was spent by the graphic artist than the program designer, it would appear, from this presentation.

The reader does not know who does what to achieve the very big impacts.

EXUBERANT USE OF COLORENHANCES UNDERSTANDING

The “Exuberant use of color enhances understanding” format strikes the eye with many bold, bright colors.

Appendix A:

Neighborhood Crime Prevention Councils–Theory of ImpactCity of Oakland

Governance:Elected positions

Rules, etcMinutes

NCPCsgrow

NSC helps residents access City

resources & staff

Build strong grassroots

organizations

NCPC members implement action on

goals

Identify 3 priorities monthly

NCPC & OPD strategize how to address priorities

CMs learn civic engagement skills

Coordinate safety trainingsNCPC

subcommittees work w/ City agencies to

impact problems

NSC & CMs inform police,

SDS, other agencies about

community priorities & needs

Reduction of Neighborhood

Problems

Residents empowered to

act on their own behalf

Concrete changes in community

Identify leaders

Beautify & improve neighborhoods

Door-to-door canvas of

neighborhood

Rev. 3/07

Communication: Outreach

Keep in touch w/ CMs

Inform City agencies

NCPC: strong grassroots org that reflects interests of community

CMs lead mtgs

City agencies are responsive to community

Train & mentor CMs

Mtg facilitationAgenda setting

OutreachCritique projects

Strategic Planning

Linking w/ City

institutions

Neighborhood & Merchant Watch inquiries about neighborhood

issues

Put up flyers in beat/ general

outreach

1:1 meetings w/ residents

Block level issues arise

Respond to phone & email inquiries

about neighborhood

issues

Residents gain info

about City resources

CMs learn to solve problems

Concrete Changes

Legend: The color of the box indicates the goal. If a box has writing in another color, it serves two goals.

NSC–Neigh Service CoordinatorCM-Community MemberSDS-Service Delivery TeamOPD-Oakland Police Dept.

Empower residents – leadership & skills

Links to City institutions

NCPC meetings

CMs know who to contact for help

CMs organize events

EXUBERANT USE OF COLORENHANCES UNDERSTANDING


The eye is drawn to the color, and not the meaning.

This format does not print well in black and white, and much critical meaning is lost without the color.

TIPS ON WRITING A LOGIC MODEL

1. A logic model should be easy to read and to understand by stakeholders who are only somewhat familiar with your program.

2. Make it clear who does what and who receives what. Subject – verb – object

3. Decide which columns your model needs: Resources/inputs may not be necessary; Activities and outputs may be duplicative.

4. Achieve a balance between showing enough detail and not too much detail.

OUTCOMES FOR DIFFERENT TYPESOF PROGRAMS

Direct contact; Social marketing; Advocacy/Policy programs

SHORT MEDIUM LONG-TERM

Changes in:[Head, heart, and gut]

• Awareness• Knowledge• Attitudes• Opinion• Aspirations• Motivation• Behavioral intent

Changes in:[Actions]

•Behavior •Skills

Changes in:[Well-being]

HealthEducationFinancial stabilityRelationships (including with offspring)Legal conditionAddiction condition

PROGRAMS WITH FACE-TO-FACE CONTACT

Activities Units of Services / Pop Reached Changes in

Knowledge / Attitudes

Changes in Skills / Behaviors

Impact / Long-Term Outcomes

Provider agencies serve a total of 74 high risk men age 18 to 24 years with wraparound services, 20 of whom are new this fiscal year. ***

54 continuing men in wraparound services.

Partners gain awareness of the couple’s inherent strengths, and are able to identify informal supports in their community.

Partners and clients increase knowledge of the social, emotional and physical needs of young men.

Partnership functioning increases (e.g., housing, finances; and decreased conflict and isolation).

Young men at risk are identified early.*

Men receive necessary services and support to reduce the effects of risk-taking behaviors and chronic mental health conditions on their relationships with partners and family.*

Men at risk remain in supportive relationships.

Partners have healthy relationships with the men with mental health and

20 new men enroll into wraparound services

Wraparound facilitator and client’s family/ partners set goals with the client and works with the client until goals are met.

360 facilitated client-partner meetings.

Instances of client risk-taking (e.g., drug abuse and unprotected sex ) are reduced.

Client’s problematic behaviors decrease.

Partners are available and

65 men continue to receive mental health services ( )

Partners of men


Changes in:

Awareness of campaign

Understanding of message

Changes in:

Attitudes

Opinion

Aspirations

Motivation

Behavioral intent

Changes in:

BehaviorsGroup norms

HealthEducationFinancial stabilityRelationships (including with offspring)Legal conditionAddiction condition

SOCIAL MARKETING PROGRAMS

The VERB™ Campaign Logic Model: A Tool for Planning and EvaluationMarian Huhman, PhD, Carrie Heitzler, MPH, and Faye Wong, MPH, RDPreventing Chronic Disease v.1(3); Jul 004http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1253476/


Media contacts

Messages and materials developed

Coalition developed

Media reps come to our org for back story

Media coverage of issues

Coalition builds capacity and skills

Regulations or Laws passed and enforced

POLICY AND ADVOCACY PROGRAMS

http://www.planning.continuousprogress.org/

POLICY ADVOCACY PROJECTAudience Inputs Activities /

TacticsOutcomes/ Benchmarks

Goals/ Impact

Elected officials

Problem assessment

Conduct issues analysis and research

Political will:2 members of Congress draft bill

Policy Development

Health officials Strategy assessment

R’ship building w/ health officials

Health officials issue statements

to Draft bill, get into Committee

Immigration officials

Message development

Organize rallies & marches

Immigration officials increaseawareness

Get bill to floor for vote

CHOICE OF MODEL FORMATS

Support $

Staff

Com’ty Liaison

s


CENTRAL

EAST

WEST

Support members

Conduct Advocacy

and outreach to “spread

First 5 message”

Organize Community

Fairs


classes


Members gain

leadership skills

Members gain

advocacy skills

Classes offer fun,

affordable and safe activities

Fairs host resource tables to inform

parents of resources



Parents increase



Children join activity class (or) Children put on wait

list

Other orgs see the need for classes

and partner to fund new

ones


Parents get info on

importance of physical

activity and to incorporate nutrition into

diet

Children become


Reduced Childhood

Obesity

Lifelong activity

Parents support children’s active

living and

sports


Recruit new members

Legend

Document,

assess, or

evaluate

Rev. 08-10-08

Problem to address Poverty and violence along with limited access to healthful foods and physical activities for 14,000 children and youth in the Valley Hi community.

Program Goals 1) Continue to create supports and opportunities for youth so that they can realize their full potential and successfully transition into adulthood, using youth‐lead coalition.2) Implement a community garden for youth and adults to connect through participation in the Coalition.

Collaborative Partners

Elk Grove Unified School District; People Reaching Out; Valley Hi Youth Adult Coalition, City of Sacramento; Hands On Sacramento; Parks and Recreation

Resources Activities Outputs Outcomes Impact

Tami Sisswa, EGUSD Liaison btwn EGUSD and Coalition 6% FTE

Erika Ergaos, People Reaching Out.Integrating youth and community components of the coalition, cultivate community partners and evaluation / reporting activities. 20% FTE + 75% FTE in‐kind

People Reaching Out Ex. Dir; Liaison with staff, EGUSD and Coalition10% FTE

New Position. Youth and Community Specialist at People Reaching OutPlanning and conducting on‐site program activities; recruit and train high school students75% FTE

Coalition Facilitate monthly VHYAC meetings and action teams.

Regular coalition meetings and action team activities.

Increased community outreach and sharing of resources.

Institutional / community practice and policy changes for improved nutrition and physical activity.

Advocacy Utilize the needs assessment to create a list of potential advocacy opportunities.

Roundtable discussion on potential advocacy opportunities.

Coalition engages in advocacy.

Youth Dev. Recruit additional youth leaders from Samuel Jackman and Valley High School.

Increased number of youth participating in Coalition.

Youth are engaged as leaders in the project. Adults support youth.

Youth Dev. Facilitate youth training retreat annually.

Number of youth who attend a retreat.

Youth have leadership skills and are prepared to be leaders and members of the VHYAC.

Youth Advocacy Train youth in advocacy information and strategies.

List of topics of trainings, number of youth attending.

Youth have advocacy skills and recognize opportunities to advocate for changes in their community.

Youth advocate for increased access to healthful food.

Youth Advocacy Youth develop advocacy and education projects.

Youth coalition members create a presentation for elementary school youth on the benefits to leading a healthy lifestyle.

Schools enable youth coalition members to make a presentation at school.

Elementary school youth become knowledgeable about the benefits of a healthy lifestyle and have access to healthful food.

Youth create a video highlighting healthy food preparation options for youth distributed in the community and potentially within the EGUSD.

Schools show the video.

Community residents and school youth become knowledgeable about healthful food preparation options.

Impact Model Logic Model

STOP Act Logic Model

CHOICE OF FORMAT: COLUMNS

• What is in 1st column?– Problem, Resources

• Activities or inputs?• Activities & outputs?• Outputs or Services/Reach?

• Choices in format of outcomes (next slide)

Fresno

MHT

Fresno

Activities Changes in knowledge /

attitudes

Changes in skills /

behaviors

Impact / Long-term outcomes

Activities Short-term outcomes

Intermediate outcomes

Long-term outcomes

Impact

Activities Outcomes: 1 year

Outcomes; 3 yrs

Outcomes : 10 yrs

Activities Change in org. group

Change in wider

coalition

Change in people reached

Change throughout community

USING LOGIC MODELS FOR EVALUATIONAND PRESENTATION

Third Quarter

Goals and Obj / Impact Model

Improved hazard management

Program provides training and standards development

Attendees learn about hazards management Attendees

promote hazard reduction at the workplaceAttendees value

the use of standards for hazard reduction

Attendees gain skills in how to adapt standards to their organization

Attendees garner necessary resources for hazard management

Staffdoessomething

People change in Intervention

This results in changes in their lives, beyond the program

Ultimately leads to achieving the goal of program

paleo

Outcome ObjectivesWhen Standard Who Change what Outcome

By.... quantity of the.... will… what noun

date how many clients increase ability to... ~~~# weeks what % individuals decrease skills for... ~~~# sessions children maintain knowledge of... ~~~age older persons reduce confidence in... ~~~

neighborhoods improve likelihood of... ~~~agencies develop incidence of... ~~~families ensure understanding of... ~~~





















Partners of men

Changes in Knowledge / Attitudes Changes in Skills / Behaviors





Partners are available and attentive to the client’s needs.

Partners of men receiving wrap learn about positive support practices.

Partners and family members have appropriate goals expectations for the client.

OUTCOMES AND OUTCOME OBJECTIVES

By June 30, 2012, 75% of the partners of clients will increase their awareness of the couple’s strengths.

By June 30, 2012, 75% of partners of clients have knowledge of positive support practices.

By June 30, 2012, 50% of clients reduce their identified risk-taking behavior.

SHOULD WE STATE THE EXPECTEDAMOUNT OF OUTCOME / OBJECTIVE?

Outcome objectives should include a benchmark or expected amount of change.A minimum of 75% of the estimated 45 participants in the

prevention services program will choose infant feeding options that minimize risk of transmission through breast feeding.

Opinion varies among people who use Logic Models about whether to put the expected amount of change for outcomes in the Logic Model. I recommend not to.

Parent Education Example: Evaluation questions, indicators

To what extent is stress reduced?To what extent are relations improved?

To what extent did behaviorschange? For whom? Why? What else happened?

To what extent did knowledge and skills increase? For whom? Why? What else happened?

Who/how many attended/did not attend? Did they attend all sessions?Supports groups? Were they satisfied – why/why not?

How many sessions were held? How effectively?#, quality of support groups?

What amount of $ and time were invested?

# Staff$ used# partners

# Sessions held Quality criteria

INDICATORS

#,% attended per session Certificate of completion

#,% demonstrating increased knowledge/skillsAdditional outcomes

#,% demonstrating changesTypes of changes

#,% demonstrating improvementsTypes of improvements

Staff

Money

Partners

Parents increase knowledge of child dev

Parents better understand their own parenting style

Parents use effective parenting practices

Improved child-parent relations

Research Facilitate support groups

Parents gain skills in new ways to parent

Parents identify appropriate actions to take

Parents of 3-10 year olds

Deliver series of 8 interactive sessions

EVALUATION QUESTIONS

Develop parent ed curriculum Reduced

stress

Parents gain confidence in their abilities

Strong families





















Partners of men






51 continued for 4+ mos.

60% of partners gained awareness of the couple’s inherent strengths, and 95% are able to identify informal supports in their community.

80% of partners and 40% of clients increased knowledge of the social, emotional and physical needs of young men.

To be determinedPartnership functioning increases (e.g., housing, finances; and decreased conflict and isolation).

Young men at risk are identified early.* Men receive

necessary services and support to reduce the effects of risk-taking behaviors and chronic mental health conditions on their relationships with partners and family.*Men at risk remain

in supportive relationships. Partners have

healthy relationships with the men with mental health and

35 men enrolled


420 facilitated client-partner meetings were held

20% of clients reduced instances of their identified risk-taking behavior.

Too soon to know:Client’s problematic behaviors decrease.

LAYERING REPORTS USING LOGICMODELS AND HYPERLINKS

Go from ppt to Word:In Word doc, at the beginning of the data, insert bookmark, name it with no spaces between letters, and click ‘add’. At the end of the data, type ‘(back)’, click on it, insert hyperlink to doc within same folder; browse to name of file, then add at end of file name: ‘#67’ (if the slide you wish to return to is number 67. Test link.In the ppt slide, click on a finding, insert hyperlink, browse to file name, and type # and the name of the bookmark. Test link

CONVERSATION AND QUESTIONS

Fourth Quarter

THANK YOU!

• You may download:• A version of these slides• Sample logic models• Sample impact models• Logic model templates

[email protected]

FURTHER RESOURCES

Slides, resources, and a recorded version of this webinar will be available on the Capacity 4 Health website.

www.capacity4health.orgScroll down to “What’s New: HIV”

Contact us for individualized capacity building assistance

Sonya Dublin, Capacity for Health415-568-3335, [email protected]

using models evaluation planning - amazon web...

Documents