how to evaluate ecosystems: the impact of engagement in program theory on organizations
TRANSCRIPT
![Page 1: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/1.jpg)
Case Studies:
The Impact of Engagement in Program Theory on Organizations
Evaluation 2015: Exemplary Evaluations in a Multicultural World
November 12, 2015
Presenter: Charles Gasper, TCC Group
![Page 2: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/2.jpg)
Think about impact
Programmatic Intervention
Programmatic Intervention
Programmatic Intervention
Programmatic Outcome
Programmatic Outcome
Programmatic Outcome
Programmatic Outcome
Programmatic Outcome
Mid-Term Outcome
Mid-Term Outcome
Mid-Term Outcome
Long-Term Outcome
Long-Term Outcome
Major ImpactEarth Shattering Impact
Evaluation
![Page 3: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/3.jpg)
Now think about evaluation
Programmatic Intervention
Programmatic Intervention
Programmatic Intervention
Programmatic Outcome
Programmatic Outcome
Programmatic Outcome
Programmatic Outcome
Programmatic Outcome
Mid-Term Outcome
Mid-Term Outcome
Mid-Term Outcome
Long-Term Outcome
Long-Term Outcome
Major ImpactEarth Shattering Impact
Evaluation
![Page 4: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/4.jpg)
Now take it one step further
Programmatic Intervention
Programmatic Intervention
Programmatic Intervention
Programmatic Outcome
Programmatic Outcome
Programmatic Outcome
Programmatic Outcome
Programmatic Outcome
Mid-Term Outcome
Mid-Term Outcome
Mid-Term Outcome
Long-Term Outcome
Long-Term Outcome
Major ImpactEarth Shattering Impact
Evaluation
![Page 5: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/5.jpg)
Or if we recognize that evaluation can have programmatic effects
Programmatic Intervention
Programmatic Intervention
Programmatic Intervention
Programmatic Outcome
Programmatic Outcome
Programmatic Outcome
Programmatic Outcome
Programmatic Outcome
Mid-Term Outcome
Mid-Term Outcome
Mid-Term Outcome
Long-Term Outcome
Long-Term Outcome
Major ImpactEarth Shattering Impact
Evaluation
![Page 6: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/6.jpg)
But what if we change our focus?
![Page 7: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/7.jpg)
What if the program is just part of the story?
![Page 8: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/8.jpg)
Organizations are multicellular creatures
Program Program
Program Program Program
![Page 9: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/9.jpg)
Not to mention part of systems
![Page 10: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/10.jpg)
And those parts affect one another
![Page 11: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/11.jpg)
Systems Approach
![Page 12: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/12.jpg)
So where are the red apples? Program
Partners
Environment
Organization
![Page 13: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/13.jpg)
So where are the red apples? Program
Partners
Environment
Organization
![Page 14: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/14.jpg)
So where are the red apples? Program
Partners
Environment
Organization
![Page 15: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/15.jpg)
Evaluation impact on organizations
![Page 16: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/16.jpg)
Rudimentary model of change for Public Media
Improved Community Perception
Improved Partner Perception
Improved Partner
Collaboration
Organizational Learning • Changes in how work is done • Changes in capacity to do the work
Our Work Increased
Awareness / Understanding
Discussion American Graduate
Champions
Increased Volunteerism &
Donations
Things Get Better for
Young People
![Page 17: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/17.jpg)
Rudimentary model of change for Public Media
Improved Community Perception
Improved Partner Perception
Improved Partner
Collaboration
Organizational Learning • Changes in how work is done • Changes in capacity to do the work
Our Work Increased
Awareness / Understanding
Discussion American Graduate
Champions
Increased Volunteerism &
Donations
Things Get Better for
Young People
![Page 18: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/18.jpg)
Documented organizational changes
1. Creation of GSD team
2. Movement to impact focus
3. Rudimentary cross-platform/department activity
4. Development of decision matrix for events
![Page 19: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/19.jpg)
Organizational process change in funding
Evaluation
Funding
Decision
Compliance
Evaluation
Learning
Evaluation
Organizational
Culture &
Interests
Financials
Interim/Final Reports
Staff Site Visits
Focus of
Learning
Individual
Program
Initiative/Funding
Stream
Portfolio
Individual
Program Learning
Initiative/Funding
Stream Focused
Learning
Portfolio
Focused
Learning
Foundation
Foundation
Focused
Learning
![Page 20: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/20.jpg)
Documented organizational changes
• Transfer to portfolio focused learning
• Staffing shifts
• Major modifications to design
• Board engagement
![Page 21: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/21.jpg)
Summary of organizational change
• Shifts in organizational focus
• Changes in:
– Staffing
– Activities
– Collaboration
![Page 22: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/22.jpg)
The results are in…
Changes in organizational culture
Efficiencies of labor
Improved quality of products / outcomes
![Page 23: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/23.jpg)
What about the partners?
• Shifts in amount and quality of collaboration
• Changes in organizational activities
![Page 24: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/24.jpg)
Partner impact?
2014 Partnerships
2015 Partnerships
![Page 25: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/25.jpg)
The results are in…
Access to underutilized resources
Improved quality of programming
Increased revenue
![Page 26: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/26.jpg)
What does this look like?
• Dedicated evaluation team member
• Collaboration with key internal stakeholders
• Organic restructuring of work
![Page 27: How to Evaluate Ecosystems: The Impact of Engagement in Program Theory on Organizations](https://reader034.vdocuments.mx/reader034/viewer/2022042619/58f1bbe31a28abb52a8b4603/html5/thumbnails/27.jpg)
Looking forward
• Is there a relationship between engagement in evaluation and organizational capacity?