Advances in Development Evaluation
InterAction Forum 2012
Session Overview
• Interactive - audience participation • Global Giving’s storytelling work• Evaluation of networks• Collaborative learning activity• Wrap up / Q&A
Introduction
• The Rockefeller Foundation Evaluation Office is supporting innovative evaluation approaches, new methods and tools and evaluation capacity development
• This session presents the work of two of RF’s grantees who are key contributors to advances in development evaluation
SKILLS
IDEASTHE EXPERIENCE
Tell us about something that surprised you at InterAction yesterday.
What is your story about? Place it somewhere inside the triangle, or in the “other” box.
OTHER
GlobalGiving Last Week
Catalyze Marketplace of $, Info, Ideas
Accessible to Growing # of More Effective Orgs
Democratizes Aid and Philanthropy
GlobalGiving Impact Framework
Design Model: Yelp! Changes the Market
Feedback makes it safe (and satisfying) to give to riskier (but maybe better performing) orgs.
Feedback identifies great NGOs and helps them learn.
Resulting data is useful for other funders, NGOs, government agencies, and researchers.
Closing the Broken Feedback Loop
Methodological Logic
Story Capture
Self-Signification
Visualization (Theme, Geo, Org, Time)Discussion
Act on InsightsMany
Narratives
Contextualized Statistics
Emergence
What’s working?
Working! • Real-Time Decision-Making Aid for GlobalGiving• Story Patterns Illuminate Needs• Actionable Constituent Voice
Promising!• Performance Dashboard for NGOs• Methodology Recognized as Useful• Interoperability with other Monitoring Tools
Working on it!• Globally Scalable Collection Infrastructure• Market-Driven Incentives for Story Collection• Open-Source Analysis Tools
strategies to achieve social impact
Jared RaynorDirector of Evaluation
InterAction ConferenceMay 2012
Evaluating Coalitions and Networks
Strategies: Outcomes: Impact:Project Inputs/ Resources:
Logic Model: Advocacy Initiatives
Issue analysis/ research
Media advocacy/ Public awareness
raising
Grass roots organizing
Coalition building/ networking
Policy analysis/ research
Legal action
Assure good policy implementation (Administrative/
Regulatory oversight, TA,
monitoring, etc.)
Lobbying and direct policy-maker
influence
Imp
rov
em
en
t in
th
e Q
ua
lity
of
Liv
ing
fo
r th
e C
om
mu
nit
y
Better defined and framed problems
Policy decisions effectively implemented
Policy makers adopt new policies based on the agenda / do not adopt harmful policies
Issues more clearly set on the public agenda and gain prominence (momentum, interest, awareness, etc.)
Impact of new policies is evaluated
Advocacy Staff: • Time, experience and
expertise (administrative, legislative, election-related and legal)
• Core skills (analytical, communication, research, etc.)
Funding:• Monetary resources• Non-monetary/ non-
staff resources (e.g. volunteers, in-kind donations)
Organizational Reputation:• Advocacy related• General perception as
a quality organization• Representative for
base constituency
Network:• Policy makers• Partner organizations• Media• Mobilized base
(C) TCC Group 2010
Ch
an
ge
in S
oci
al S
tru
ctu
re R
efle
ctin
g P
osi
tive
So
cia
l Ju
stic
e S
hift
Co
ali
tio
n/N
etw
ork
(a
s a
n A
cto
r a
s co
mp
are
d t
o a
st
rate
gy)
No
np
rofi
t o
r A
dv
oc
ac
y
Org
an
iza
tio
n (
use
s co
alit
ion
as
a s
tra
teg
y)Acting
Organization:
Continuum of Inter-Organizational Relationships (IORs)
Mutual Accountability
SeparateAccountability
InformalOrganization
Formal Organization
Informal Network
Coalition
Collaboration
Strategic Alliance
Joint Venture
Merger
Committee
Formal Network
Adapted from: Wolf, T. (2003). A Practical Approach to Evaluation of Collaborations. In T.E. Backer (Ed.). Evaluating Community Collaborations. New York, NY. Springer Publishing Co. with additional acknowledgement of K. Hobson and M. Hightower King at AEA 2009.
Relationships for What Purpose?
• Respond to complexity• Cope with turbulence/complexity• Acquire resources (while maintaining autonomy)• Decrease transaction costs• Gain legitimacy/power
Positive
Externalities(Value
)
Negative
Externalities (Congestion)
What Are We Measuring?
Coalition
Individual
An Entanglement of Relations
Coalition
Individual
Organization
Individual
Organization
Individual
Organization
Individual
Number of Relationships
2 Person Coalition: 83 Person Coalition: 184 Person Coalition: 32
A Simplified Approach for Evaluation
Member Capacity
Network/ Coalition Capacity
Outcomes
Image credit: http://www.cugelman.com/research/united-nations-web-network.htm
Member Capacity
Member Capacity
• What do you want to get out of the network/coalition? • How do you justify your involvement?• How much time can you devote?• What can you offer?
Individual Capacity
Network Coalition
Boundary spanners Skill/knowledge to work collaboratively
Time/space to engage in network Time and Commitment to working in coalition
Relevant skills and interest Relevant decision-making authority
Desire to be part of something bigger
Strategic use of coalitions to fill critical gaps and leverage resources toward achieving your mission
Coalition/Network Capacity:A Framework
The Critical Organizational Capacities for Advocacy Initiatives: The Logic Model “Inputs” Initiative Logic Model
Imp
rove
men
t in
th
e Q
ual
ity
of
Liv
ing
fo
r th
e C
om
mu
nit
y
Issue analysis/ research
Media advocacy/ Public awareness raising
Grass roots organizing
Coalition building/ networking
Policy analysis/ research
Legal action
Administrative/ Regulatory implementation influence
Lobbying and direct policy-maker influence
Management:
Adaptability:
Technical:
Access window of
opportunity/ threat:
•Offense•Defense
Org. Culture
(C) TCC Group 2008
Leadership:
A Few Key IOR Capacities
Goal Destination
(What is the IOR trying to achieve?)
Value Proposition
(Why is this IOR the right approach?)
• Rules, procedures and decision-making• Action/Exchange-oriented• Leadership
Entropy
(disorder)
Bureaucrac
y(rigidit
y)
Multiple Levels of Outcome
Community
Individual
Benefits to Individuals
Community Impact/
OutcomesDevelopment
of IOR
Benefits to Organizations
Capturing ‘Externalities’
• Better implementation/coordination of actual programs addressing the issue (More entrenched programming)
• Cross-fertilization and Innovative ideas (Group think)
• Other specific project goals met
• Sustained networks/relationships (Relationships hurt)
• Reduced sense of isolation (Personality conflicts)
Outcome questions for IORs in Emergency Settings
• Has the IOR increased the ability to coordinate during varying stages of the emergency, including preparation?
• Did the IOR reduce political complexity?
• Did the IOR reduce or increase time and resource constraints?
• Has the IOR increased the reliability of information available?
• Has the IOR facilitated overcoming infrastructure challenges?
• Has the IOR enhanced the well-being of personnel?
• Has the IOR facilitated better relationships with affected populations and donors?
• Has the IOR increased ability to engage with the media?
Conclusion
Contact Information
Jared RaynorDirector of Evaluation
TCC Groupwww.tccgrp.com
Leveraging the Evaluator
• Navigate power differentials• Use to:
• Build (inform development) • Sustain (strengthen existing
performance; identify inefficiencies)
• Reflect (accountability for resources and prep for future)
• Understand how to position yourself in a network
• Help articulate goal destination and value proposition
Mobile Phone Principles of Good IOR Behavior
Getting the Right Plan
• What do you need to accomplish?
• Do you have appropriate bandwidth (value vs. congestion)?
• Right amount of minutes/airtime?
1
Mobile Phone Principles of Good IOR Behavior
Aware of Surroundings• How should others contact
you? You contact them?• Are you talking too loudly?
Too quietly?• How much background
noise is there?
2
Photo source: whatisacellphonejammer.com
Mobile Phone Principles of Good IOR Behavior
3Strategic Use• Do you have the right
connections?• Are there new apps?• Are you playing too much Angry
Birds?• Rate/assess your experience
Mobile Phone Principles of Good IOR Behavior
Aware of Surroundings• How should others contact
you? You contact them?• Are you talking too loudly?
Too quietly?• How much background
noise is there?
Getting the Right Plan
• What do you need to accomplish?
• Do you have appropriate bandwidth (value vs. congestion)?
• Right amount of minutes/airtime?
Strategic Use• Do you have the right
connections?• Are there new apps?• Are you playing too much Angry
Birds?• Rate/assess your experience
21
3