karen horsch evaluating change v.2

Evaluating ChangeKaren Horsch

Evaluation Consultant

We are experiencing epidemic rates of childhood obesity that have long-term consequences for health and well-being of individuals and society

It’s getting worse, not better

What we know:

Reversing this trend in a sustained way is going to take fundamental changes to the way we think and act: we cannot program our way out of this

Reversing this trend in a sustained way is going to require that we build a movement

What we know:

We need to shift perceptions such that physical activity and healthy eating are the social norm

We need to establish environments that make individuals’ default decisions healthy

What needs to happen:

They raise issues of urgency (identify problems)

They build a base of interested and motivated individuals and coalitions

They educate and raise awareness They advocate for positive changes in

policies and institutions They identify and promote promising

interventions and programs (show what’s possible)

What movements do….

How does evaluation fit in?

Data about the movement (is the movement moving?)

Data to build the movement

Two levels of evaluation when building a movement:

So how do we evaluate whether we are building a movement?

NOT: What did we accomplish?

BUT: Did we accomplish what we set out to accomplish? Why or why not?

Start with the right evaluation question:

The “theory of change” about how to build a movement

Laying out what we hope to accomplish:

The Strategies and Interventions

The Results we hope to achieve:

• Short-term

• Long-term

• Intermediate term

Strategies

• Coalition building

• Grassroots organizing/mobiliz-ation

• Leadership development

• Advocacy training

• Media advocacy

• Identification & replication of proven programs

Short-Term Outcomes

• Increased awareness about and prioritization of the issue

• Strong and diverse network of advocates

• Strengthened alliances

• Increased media attention to the issue

• Established relationships with key decision-makers

.

Intermediate-Term Outcomes

• Increased public and policymaker support for policies and actions that promote positive changes in the issue

• Actions supporting physical activity and healthy eating:

-- more funding-- more programs-- policy change-- changes in institutions

Long-Term Outcomes

• Behavior changes: increased physical activity and healthy eating and reduced screen time

• Reductions in obesity rates

• Improved health

• Who the decision makers are• Who the public is• Who partners might be

Being clear about:

• Information sent out and overall message framing

• Number and type of people who receive information

• Attendance at meetings/events• Website hits• Public opinion surveys

Increased awareness about and prioritization of the issue:

• Number and type of people trained in advocacy

• Number and type of people taking on leadership roles in advocacy

• Number of new advocates • High profile people who adopt the issue and

publicly advocate for it• Actions taken by advocates

Strong and diverse network of advocates:

Number and type of coalitions/partnerships established

Number of active members of coalition Health of coalitions Number and type of cross-organization and

cross-sector partnerships/collaborations Network analysis

Strengthened alliances:

Number of press releases Number of op-eds/letters to the editor

published Number of articles covering the issue Relationships with the press How the issue is framed in media

Increased media attention to the issue:

Visits to decision makers Number of decision makers attending

events related to issue % of phone calls returned by decision

makers

Established relationships with decision makers:

Number of decision makers and/or opinion leaders who express support for the issue

Policy makers’ press releases and comments in legislative hearings

Policy maker attendance at hearings related to the issue

Increasing support for:

New policy proposals New policies passed Policies expanded/or not reversed Implementation of policies

Improving Policies:

Balancing the data collection effort with the utility of the information collected

It’s not all countable: the importance of qualitative data

Data Considerations

So what data do we need to build the movement?

Obesity as connected to environments and not solely individual behavior

Role of government in this Importance of prevention rather than long-

term health consequences

Data that helps to reframe the issue:

Population-level data collection Community mapping and other visuals The cost of doing nothing

Data that points to the urgency of the issue and its causes:

What programs work and how well◦ How many they serve◦ The outcomes they achieve

What programs cost or save

Data about what works (Program Evaluation)

“Policy change occurs when community leaders receive credible and reliable data and research AND community members provide personal stories and advocate for change.”

-- Kay Monaco, New Mexico Voices for Children

Policymakers receive a lot of information Easy visuals: mapping, metrics, summaries Personal stories and testimonials matter Timeliness and credibility are critical

Policymakers as Audience

POLICY CHANGE

Organizing and Coalition

Building

Finding information/ Research

Making a Plan

Mobilization and

Communications

Climate ready

For change

Source: PolicyLink