how to evaluate ecosystems: the impact of engagement in program theory on organizations

Case Studies:

The Impact of Engagement in Program Theory on Organizations

Evaluation 2015: Exemplary Evaluations in a Multicultural World

November 12, 2015

Presenter: Charles Gasper, TCC Group

Think about impact

Programmatic Intervention

Programmatic Intervention

Programmatic Intervention

Programmatic Outcome

Programmatic Outcome

Programmatic Outcome

Programmatic Outcome

Programmatic Outcome

Mid-Term Outcome

Mid-Term Outcome

Mid-Term Outcome

Long-Term Outcome

Long-Term Outcome

Major ImpactEarth Shattering Impact

Evaluation

Now think about evaluation

Programmatic Intervention

Programmatic Intervention

Programmatic Intervention

Programmatic Outcome

Programmatic Outcome

Programmatic Outcome

Programmatic Outcome

Programmatic Outcome

Mid-Term Outcome

Mid-Term Outcome

Mid-Term Outcome

Long-Term Outcome

Long-Term Outcome

Major ImpactEarth Shattering Impact

Evaluation

Now take it one step further

Programmatic Intervention

Programmatic Intervention

Programmatic Intervention

Programmatic Outcome

Programmatic Outcome

Programmatic Outcome

Programmatic Outcome

Programmatic Outcome

Mid-Term Outcome

Mid-Term Outcome

Mid-Term Outcome

Long-Term Outcome

Long-Term Outcome

Major ImpactEarth Shattering Impact

Evaluation

Or if we recognize that evaluation can have programmatic effects

Programmatic Intervention

Programmatic Intervention

Programmatic Intervention

Programmatic Outcome

Programmatic Outcome

Programmatic Outcome

Programmatic Outcome

Programmatic Outcome

Mid-Term Outcome

Mid-Term Outcome

Mid-Term Outcome

Long-Term Outcome

Long-Term Outcome

Major ImpactEarth Shattering Impact

Evaluation

But what if we change our focus?

What if the program is just part of the story?

Organizations are multicellular creatures

Program Program

Program Program Program

Not to mention part of systems

And those parts affect one another

Systems Approach

So where are the red apples? Program

Partners

Environment

Organization

So where are the red apples? Program

Partners

Environment

Organization

So where are the red apples? Program

Partners

Environment

Organization

Evaluation impact on organizations

Rudimentary model of change for Public Media

Improved Community Perception

Improved Partner Perception

Improved Partner

Collaboration

Organizational Learning • Changes in how work is done • Changes in capacity to do the work

Our Work Increased

Awareness / Understanding

Discussion American Graduate

Champions

Increased Volunteerism &

Donations

Things Get Better for

Young People

Rudimentary model of change for Public Media

Improved Community Perception

Improved Partner Perception

Improved Partner

Collaboration

Organizational Learning • Changes in how work is done • Changes in capacity to do the work

Our Work Increased

Awareness / Understanding

Discussion American Graduate

Champions

Increased Volunteerism &

Donations

Things Get Better for

Young People

Documented organizational changes

1. Creation of GSD team

2. Movement to impact focus

3. Rudimentary cross-platform/department activity

4. Development of decision matrix for events

Organizational process change in funding

Evaluation

Funding

Decision

Compliance

Evaluation

Learning

Evaluation

Organizational

Culture &

Interests

Financials

Interim/Final Reports

Staff Site Visits

Focus of

Learning

Individual

Program

Initiative/Funding

Stream

Portfolio

Individual

Program Learning

Initiative/Funding

Stream Focused

Learning

Portfolio

Focused

Learning

Foundation

Foundation

Focused

Learning

Documented organizational changes

• Transfer to portfolio focused learning

• Staffing shifts

• Major modifications to design

• Board engagement

Summary of organizational change

• Shifts in organizational focus

• Changes in:

– Staffing

– Activities

– Collaboration

The results are in…

Changes in organizational culture

Efficiencies of labor

Improved quality of products / outcomes

What about the partners?

• Shifts in amount and quality of collaboration

• Changes in organizational activities

Partner impact?

2014 Partnerships

2015 Partnerships

The results are in…

Access to underutilized resources

Improved quality of programming

Increased revenue

What does this look like?

• Dedicated evaluation team member

• Collaboration with key internal stakeholders

• Organic restructuring of work

Looking forward

• Is there a relationship between engagement in evaluation and organizational capacity?

CONTACT US @TCCGRP #EVAL15

Charles Gasper cgasper@tccgrp.com

@KarcsiG