measuring results that matter: evaluating ced impacts
Post on 18-Jan-2016
31 Views
Preview:
DESCRIPTION
TRANSCRIPT
Measuring Results That Matter: Evaluating CED Impacts
The Community Economic Development Technical Assistance Program (CEDTAP)
Presenters:
David DriscollChair, CEDTAP Advisory Panel
Edward JacksonDirector
Gail ZbochNational Coordinator
Ray FunkPrairies Regional Coordinator
Colleen KastingBC Regional Coordinator
3
4
Measuring Results That Matter: Evaluating CED Impacts
Overview of session:
1:30-4:00
CEDTAP at-a-glance CED knowledge clusters Results to date Evaluating CED impacts: assessing the CEDTAP
portfolio Group discussion Sharing preliminary results of impact surveys Next steps
5
Goal, Objectives and Methods
Goal Demonstrate the effectiveness and legitimacy of community
economic development as a strategy for economic and social change
Objectives Strengthen the capacity in CED of community-based
organizations Strengthen the capacity of the CED sector as a whole Methods Grant-making to CEDOs to access technical assistance Building of a pool of TA Providers Grant-making to sector organizations Knowledge production and dissemination Participation in research and policy alliances and consortia
6
CEDTAP At-a-glance
3CI
CEDTAPNational Secretariat
Donors:J.W. McConnell Family Fdn Bell Canada The Ontario Trillium FdnWED, DEC-Québec Other Corporations
Advisory Panel
CED Organizations
(CEDOs)
Technical Assistance Providers
CEDTAP Regional Coordinators
7
Program Phases, 1997-2008
Phase I pilot phase (1997-2000) $3M Some 90 grants, 30 TA Providers, three national conferences Funder: McConnell Foundation Phase II expansion phase (2001-2005) $7M Some 330 grants, 800 TA Providers, three national
conferences, policy roundtables Funders: McConnell Foundation, Bell Canada, The Ontario
Trillium Foundation, DEC Québec, WED-BC, WED-Prairies, RBC, Power Corp., Tides Canada
Phase III transition phase (2006-2008) $3M+ Grant-making, knowledge mobilization, corporate
engagement, research alliances Funders: Bell Canada, The Ontario Trillium Foundation,
McConnell Foundation, WED-BC, WED-Prairies, DEC-Québec, SSHRC, Power Corp., RBC, others.
8
20 CED Knowledge Cluster Areas
• Agriculture & Fishing• Arts & Culture• Disability and Mental Health-
Consumer Survivor Businesses
• Community Land Trusts• Community Economic
Renewal• Community Tourism• Employment Strategies• Environmental Management
& Enterprises• E-Strategies
• Financial Equity• Food Processing & Food
Security• Health & Nutrition• Homelessness & Poverty• Individual Development
Accounts• Industrial Re-conversion• Forestry• Sustainable Housing• Women Entrepreneurs• Youth Entrepreneurship
9
Community Strengthening Projects Categorized by CED Cluster (2001-2005)
0
5
10
15
20
25
30
35
40
Comm
unity
Econo
mic
Renew
al
Comm
unity
Land
Tru
sts
Comm
unity
Loan
Fun
ds
Comm
unity
Tour
ism
Disabi
ltiy &
Men
tal H
ealth
Cons
umer /
Sur
vivor
Bus
iness
es
E-stra
tegie
s
Emplo
ymen
t Stra
tegie
s
Enviro
nmen
tal M
anag
ement
& E
nter
prise
s
Financia
l Equit
y
Food
Proce
ssing
/ M
arke
ting
(inclu
ding
food
ban
ks)
Fores
try
Health
& N
utrit
ion
Homel
essn
ess &
Pov
erty
Indi
vidua
l Dev
elop
men
t Acc
ount
s
Indu
stria
l Rec
onve
rsio
n
Susta
inab
le H
ousin
g
Wom
en E
ntre
pene
urs
Youth
Ent
repr
enur
ship
Servic
es
10
Results: British Columbia, 2001-05
Top CED clusters: Community economic
renewal Employment strategies Environmental
management Food
processing/marketing
Current activities: Bell/McConnell WED support for regional
coordination Partnership with Vancity
Community Foundation
0
5
10
15
20
2001 2002 2003 2004 2005
$564,000 granted to 49 projects
11
Potluck Café Society Inc.
Potluck's Flash Frozen Gourmet Meals Project
12
Results: Prairies, 2001-05
Top CED clusters: Agriculture & fisheries Community economic
renewal Disability and mental
health-consumer survivor businesses
Current activities: Bell / McConnell WED support for regional
coordination
0
5
10
15
20
2001 2002 2003 2004 2005
$496,500 granted to 43 projects
13
Stardale Women’s Group Inc. Foundation
Stardale Artistic Co-operative
14
Results: Ontario, 2001-05
Top CED clusters: Community economic
renewal Disability and mental
health-consumer survivor businesses
Employment strategies Women entrepreneurs
Current activities: Bell/McConnell The Ontario Trillium
Foundation
05
10152025
2001 2002 2003 2004 2005
$ 762,000 granted to 76 projects
15
Good Day Workshop Inc.
Participants at the Good Day Workshop,
refinishing old furniture
16
Results: Quebec, 2001-05
Top CED clusters: Community economic
renewal Employment strategies Environmental
management Youth entrepreneurship
Current activities: Bell / McConnell DEC Québec
0
5
10
15
20
25
30
2001 2002 2003 2004 2005
$ 625,500 granted to 74 projects
17
L’atelier De Formation Socioprofessionnelle De La Petite-nation
Des participant(e)s de l’Atelier FSPN
18
Results: Atlantic Region, 2001-05
Top CED clusters: Arts & culture Community loan funds Community tourism
Current activities: Bell / McConnell
0
1
2
3
4
5
6
7
8
9
2001 2002 2003 2004 2005
$250,000 granted to 21 projects
19
Central Development Association
Promotion of social and economic development in the Northern Peninsula Central Region of Newfoundland
20
Evaluating CED Impacts:Assessing the CEDTAP Portfolio
Why evaluate CED impacts? New policy actors (eg, new federal government) are
unfamiliar with CED Increased emphasis on accountability, value-for-money
and results Heightened competition for scarce public and private
funds Need to evaluate program effectiveness and impact Opportunity to contribute to the body of knowledge on
CED policy and practice Opportunity to enable CED sector to lever bigger impacts
in high growth contexts (winter Olympics in BC; energy and resources on prairies)
21
How to Evaluate CED Impacts?
Test against program results chain for community initiatives
Utilize best-practice methods and tools from the field as a whole and the portfolio, in particular
“Slice and dice” the portfolio by knowledge cluster and policy area
Be clear about units of analysis: individual/household, group or enterprise, program/fund, CEDO, community, region
Assess progress on qualitative and quantitative indicators Survey CEDOs Survey Technical Assistance Providers (TAPs) Sample a small number of organizations and projects for
detailed case study analysis (eg, 15)
22
Results Chains
From: CEDTAP Program Summary, 2002
Results Chain for Community Initiatives
TA Initiative Outputs Organizational Outcomes Community ImpactsCoaching and Mentoring
TrainingPlanning process
AdviceTools
Evaluation Processes
New Knowledge AcquiredStronger Leadership and Vision
Business / Project PlanningManagement / SystemsEconomic Sustainability
Social Viability / LegitimacySectoral Expertise Gained
EmploymentBetter Quality of Life
Income / AssetsSocial Capital
Credibility of CEDOther Household Measures
Effective Partnerships in and Between Communities
Results Chain for Sector Initiatives
Sector Initiative Outputs Sector Outcomes Sector ImpactsTools
ResearchPolicy
DisseminationWorkshopsConferences
LearningConnectivityPartnerships
Resource UsePolicy Work
Credibility of CEDPro-CED Policy
Increased Public & Private Spending on CEDIncreased # of CED professionals
23
Evaluation: “The 21st Cluster”
Unit of analysis organization/project
Region Western Valley Development Authority, NS Evaluation Toolkit of Local Capacity
Community CIEL, Community Vitality Index
CEDO OISE/UT, Expanded Added Value Statement*PEACH, Evaluation Framework
Program Saint John Human Development Council /Tamarack institute, Small Business Program
Evaluation Framework
Enterprise Social Capital Partners, Social Return on Investment*
Households/ Eko Nomos/WEDC, Sustainable LivelihoodsIndividuals Approach*
Flavour Budzzz, Wisconsin Quality of Life Index
*Not CEDTAP project
24
Expanded Value Added Statement
Measures the economic and social value added to raw materials, products and services by non-profits and cooperatives’ use of labour and capital
A social accounting model created by Mook (see Quarter, Mook and Richmond, What Counts, 2003)
Estimates a comparative market value for non-monetized social contributions
Primary outputs refer to the value realized by the organization’s direct provision of services to advance its mission (eg. Revenues from fees for service, social or volunteer labour, services donated to the community)
25
Expanded Value Added Statement (Cont)
Secondary outputs refer to value to an organization’s members or customers realized through indirect outputs (eg. Skills training, learning through membership meetings, board and committee governance meetings)
Tertiary outputs refer to value realized by those other than the organization’s members or customers (eg. Consultations with other organizations or distribution to stakeholders)
In practice, EVAS builds credibility through the use of conservative assumptions in calculating comparative market value of social contributions
26
Key Themes in Building a Long List of Potential Case Study Cedos
Urban youth/immigrant employment Urban homelessness/poverty/redevelopment Youth entrepreneurship Alternative energy (urban) Community loan funds (urban) Women’s entrepreneurship (urban) Disability groups (urban) Mental health consumers/survivors (urban) Restaurant/catering (urban) Wood products (rural) Community tourism (rural) Community futures/development corporations (rural)
27
Exercise: Group Discussion
At your table, discuss two questions:
2. How does your organization assess CED impacts?
3. What is the biggest challenge your organization faces in assessing CED impacts?
Please record your group’s answers. You have 10 minutes for this exercise!
28
Findings in Process: Surveys of CEDTAP Grantees and TA Providers
1. Significant impacts of CEDTAP projects in which you have been involved?
2. Significant impacts of non-CEDTAP projects in which you have been involved?
3. Important indicators of: economic, social and environmental impact?
4. Key success factors that enable CED organizations to generate significant impacts?
5. Methods, tools and indicators for assessing impact that are most useful in your work?
6. Key barriers or obstacles that limit the impacts produced by CED organizations?
29
Generating Impacts: Top 10 Success Factors
Funding: start-up, core, promotional
Realistic and achievable plan
Input from stakeholders
Experienced and strong volunteer and board commitment
Commitment and participation of of management and staff
Time to plan and develop
Government, business and community support
Community partnerships
Public relations and marketing strategy
Ability to manage financial and social goals simultaneously
30
Economic Indicators
Number of jobs created Assisting people with training or starting a business Creation of social enterprises Increase in wealth – individual and community Increased spending on purchase of local supplies Decreased expenses through increased productivity,
sharing resources, eliminating wasteful practices Change of attitude and behaviour in evaluating business
from a financial measure to include the blended bottom line
Degree to which the venture can or will contribute to the overall society
31
Social Indicators
Ability to support oneself and one’s family Increase in self-esteem and self-image Community ownership of initiative Development of local leadership Improvement in community quality of life issues: safety,
housing, health, less hospitalization, recreation, transportation, childcare
Increase in the participation levels of the client group both as participants in the venture and recipients of the service
Legacy of work supported through long-term planning with government, business and not-for-profit sector
32
Notable Methods & Tools:
Formative and summative surveys
Good benchmark data socio-economic indicators that measure lifestyle & healthy community improvements
Identify targets and projections coupled with goals and objectives, all in keeping with the mission
Pre and post evaluations using qualitative and quantitative measures
Working with an outside facilitator (arm’s-length)
33
Notable Methods & Tools (Cont):
Feedback from focus groups
Communicate often and openly with stakeholders
Sharing the learning with colleagues and community: ongoing email, coffees, lunches
“….Organizations have asked for a copy of our CEDTAP-funded strategic plan”
Tamarack teleconferencing and initiating our our teleconferencing for specific projects
34
Generating Impacts: Barriers/obstacles
Lack of enough resources: financial & human
Time spent on writing grants and looking for money
Lack of interest by mainstream social service agencies to take a CED approach
Lack of long-term vision and plan for sustainability
Lack of leadership
Reliance on volunteers
35
Barriers/obstacles (Cont)
Projects too short in length Inability to engage community and stakeholders Lack of understanding by government agencies
and sponsors of the complexities of new industry development as well as “soft” changes that measure success
Success requires long-term commitment, for which some funders may not have patience
36
Measuring Results That Matter: Evaluating CED Impacts
Next steps: Additional analysis of survey data Further analysis by clusters of the portfolio Identification of case study organizations Joint research with CEDOs, technical
assistance providers, academics and funders Dissemination of findings
Thank you!
top related