using evaluation to shape and direct comprehensive ... · yasir dildar centre for community based...

61The Canadian Journal of Program Evaluation Vol. 26 No. 2 Pages 61–88ISSN 0834-1516 Copyright © 2012 Canadian Evaluation Society

Corresponding author: Rich Janzen, Centre for Community Based Research, 73 King St. W, Kitchener, ON, Canada N2G 1A7; <[email protected]>

USiNG EvAlUAtiON tO ShApE ANd diRECt COmpREhENSivE COmmUNity iNitiAtivES: EvAlUAtiON, REflECtivE pRACtiCE, ANd iNtERvENtiONS dEAliNG With COmplExity

Rich JanzenCentre for Community Based Research and Wilfrid laurier University

daniela Seskar-hencicRegional municipality of Waterloo

yasir dildarCentre for Community Based Research, Kitchener, Ontario

peter mcfaddenWaterloo Region immigrant Employment Network

Abstract: the purpose of this article is to discuss how evaluation can be used to shape and direct Comprehensive Community initia-tives in an ongoing way. it does so by offering a case example of the Waterloo Region immigrant Employment Network (WR-iEN). the article begins by reviewing how participatory action research can encourage reflective practice. After the WRiEN case example is presented, we consider this example in light of collaborative evaluation literature and five facilitators of reflec-tive practice: (a) be location-based, (b) value experiential and practical knowledge, (c) provide ongoing feedback, (d) facilitate democratic dialogue, and (e) focus on a vision for the common good. the article ends by discussing contributions to the broader evaluation knowledge base, particularly developmental evalua-tion.

Résumé : le but de cet article est de discuter comment l’évaluation peut servir à former et diriger d’une façon continue les initiatives communautaires intégrées. On vous offre, par exemple, le cas du réseau d’emplois des immigrants à la région de Waterloo (WRiEN). l’article commence par examiner comment la recher-che-action participative peut encourager la pratique réflexive. Après la présentation du cas d’exemple du WRiEN, on considère cet exemple par rapport à la littérature sur l’évaluation collabo-rative et cinq facilitateurs de la pratique réflexive : (a) se baser sur la localisation, (b) valoriser les connaissances expérientielles

62 The Canadian Journal of Program evaluaTion62

et pratiques, (c) fournir la rétroaction continue, (d) faciliter le dialogue démocratique, et (e) concentrer sur une vision de l’inté-rêt commun. Cet article se termine en discutant les contributions à la base plus large de connaissances en évaluation, en particu-lier, l’évaluation développementale.

Since the 1990s, Comprehensive Community initiatives (CCis) have emerged as a common means of addressing social con-cerns within local communities. CCis are guided by the principles of comprehensiveness and community-building (Kubisch, fulbright-Anderson, & Connell, 1998). that is, CCis are designed to address social problems holistically, at multiple levels, and with a view to pro-moting sustainable systems change (comprehensiveness), which is best achieved through the broad-based, cross-sectoral, and equitable engagement of citizens who act as the architects of their own solu-tions (community-building) (Baker, 2003; foster-fishman, Nowell, & yang, 2007). the underlying assumption is that significant com-munity improvements will not occur unless the surrounding system adjusts to accommodate the desired goals. As such, CCis not only work for changes for individuals and groups, but also seek to improve the conditions in which they are more likely to thrive (Baum, 2001; Natasi & hitchcock, 2009). this implies that community stakehold-ers (people who have a stake in the issue under consideration) are challenged to create new and different ways of working together (foster-fishman et al., 2007). it also means that CCis are designed to evolve over time so as to respond to ever-changing community needs and priorities (lafferty & mahoney, 2003).

Given increasing demands for accountability on investment, there are growing expectations that CCis be evaluated. Evaluation is seen to be important because CCis (a) benefit from the support of a wide range of constituents who need to be kept informed of progress and outcomes; (b) need to generate useful feedback to guide action within a dynamic context; and (c) hold the opportunity to contribute to the broader, longer-term “social learning” that can be transferred to other community settings (Kubisch et al., 1998). it has been widely recognized that it is difficult to evaluate CCis (foster-fishman & long, 2009; Natasi & hitchcock, 2009). With summative evaluations the challenge lies in measuring and analyzing broad multisectoral outcomes within a complex socio-political context where the inter-vention is only one factor (Kubisch et al., 1998). however, and more central to this article, formative evaluation of CCis can also be chal-lenging. formative evaluations are conducted as the intervention unfolds for the purpose of learning, shaping, and directing it in an

6363la revue Canadienne d’évaluaTion de Programme

ongoing way (Shadish, Cook, & leviton, 1991). here the challenge lies in the dynamic nature of CCis, whose flexible and developmental nature in response to shifting local dynamics adds ambiguity and uncertainty to evaluation design and implementation (Baum, 2001).

A program theory approach to evaluation is typically seen as a means of beginning to address these challenges (lafferty & mahoney, 2003). A program theory attempts to explain how an intervention is sup-posed to work, rationally linking its resources and activities with its short- and long-term intended outcomes (Chen, 2005). A program theory approach can also be useful in sharpening program plan-ning—isolating those efforts that lead to desired outcomes and those that do not (mackenzie, & Blamey, 2005). Nevertheless, a CCi’s the-ory of change is not necessarily unified or stable, but rather evolves over time in response to complex and fluid environments (Baum, 2001; lafferty & mahoney, 2003). Evaluations of CCis therefore require mechanisms to enable ongoing reflective practice—mecha-nisms that enable stakeholders to collectively reflect on what they have done (practice) and what they have learned about what was effective (theory), all for the sake of adapting and informing their future practice and deepening their program theory (Natasi & hitch-cock, 2009). Such mechanisms of reflective practice are particularly needed within evaluations of innovative interventions (as CCis often are), where previous models of practice are lacking and where con-texts are complex and fluid (as is often the case with CCis). patton (2011) recently coined the term “developmental evaluation” to dis-tinguish evaluations of these complex systems-change interventions from traditional formative evaluations that seek to improve fixed-models of practice in more stable environments.

practical examples of evaluations of CCis that use a theory of change approach are beginning to emerge in the evaluation literature. how-ever, these typically involve summative evaluations that report on outcomes and related methodological challenges (e.g., foster-fish-man & long, 2009; mackenzie & Blamey, 2005). there are limited examples demonstrating how evaluation can be used to shape and direct CCi practice in an ongoing way. the purpose of this article is to speak to this gap by discussing the processes and mechanisms of how evaluation can be an avenue to spur reflective practice within CCis. it does so by offering a case example of an evaluation of one CCi (the Waterloo Region immigrant Employment Network—WRiEN) that adopted a participatory action research approach. the article begins by first reviewing the broader literature of how a participatory ac-tion research approach can facilitate reflective practice. following


the case example, we bring evaluation literature into the conversa-tion and link WRiEN’s reflective practice to this literature. in this way, the article bridges the gap between two long-standing bodies of literature (collaborative evaluative inquiry and participatory ac-tion research) that often remain in parallel. We conclude by briefly discussing how this article contributes to the broader evaluation knowledge base, including that of developmental evaluation. We note how the case study illuminates that the formative-developmental distinction may be less categorical than patton (2011) presents, but can rather be seen as a continuum in how loosely or tightly a par-ticular program theory is held.

pARtiCipAtORy ACtiON RESEARCh ANd REflECtivE pRACtiCE

As one type within the broader typology of research, evaluation can draw on the growing literature that discusses how research can stimulate reflective practice. Such approaches to research operate under a variety of names, including action research (e.g., Stringer, 2007), community-based research (e.g., travers et al., 2008), commu-nity-based participatory research (e.g., israel, Eng, Schulz, & parker, 2005; minkler & Wallertsein, 2008), or—the preferred term in this article—participatory action research (e.g., Kemmis & mctaggart, 2005). Regardless of label, this approach fuses the “northern” utiliza-tion-focused action research tradition initiated by Kurt lewin (1948), with the “southern” emancipatory participatory tradition initiated by paulo friere (1970) (for fuller discussions of these historical roots, see Cargo & mercer, 2008; Wallerstein & duran, 2003). Common to these traditions is a commitment to interweave theory and practice within a research process in a way that leads to knowledge-driven collective action. indeed, “learning as we go” is a key value of partici-patory action research (Nelson, Ochocka, Griffen, & lord, 1998)—a value fitting well with the evolving and responsive nature of CCis.

Within participatory action research, theory and practice are inter-connected through a process known as the reflective action cycle. the cycle generally includes some combination of planning, action, and reflection in successive spirals over time (lewin, 1948; Stringer, 2007; Wallerstein & duran, 2003). Within the process of research (including evaluation), the reflective action cycle could be conceived as four nonlinear and repeated phases (laying the foundations, plan-ning, information gathering/analysis, and acting on findings) that are ever-attuned and adaptive to an emerging context and ongoing learning (see figure 1). While we elaborate on these phases in more


detail elsewhere (Janzen, hatzipantelis, vinograd, & Kellerman, 2004; taylor & Botschner, 1998), it is worth highlighting here that the phases emphasize not only traditional technical elements of research, but also foreground the relational aspects of collaborative research. they do so because of a belief that a collaborative process of inquiry is as important as the outcomes or findings of the research (Reason, 2006). Research produces not only a vision for future collec-tive action, but also builds a sense of community and inspires people to work together toward a common goal (Stringer, 2007).

Figure 1The Four Phases of Participatory Action Research

Adapted from CCBR, 1998; 2004.

the broader research literature also provides insights into how reflec-tive practice can be facilitated through research. Below we outline five facilitators that we gleaned from the literature and that represent our theoretical contribution to the participatory action research dis-course. While we do not presume these facilitators to be exhaustive, they do provide a means of connecting participatory action research to the knowledge base of evaluation (discussed later in the article).


Be location-Based

the first facilitator of reflective practice is to focus the scope of research to a particular community setting. Being location-based requires a deep ecological understanding of the local context (New-brough, 1995). it also implies that design changes to respond to a context, which is dynamic in nature (Reason, 2006). this is because the emphasis is on solving local problems for a specific group of people—local theory for local practice (Stringer, 2007). Reflecting on local practice is pre-eminent, as generalizing theory to benefit other settings is of secondary importance. Within the evaluation of CCis, being location-based implies a clear bounding of what constitutes the intervention under study and what constitutes external context (foster-fishman et al., 2007). it also implies that depictions of a pro-gram’s theory of change, such as program logic models (mclaughlin & Jordan, 2004), are not viewed statically but change and evolve in response to the deepened understanding of what kind of intervention is needed with a specific location.

value Experiential and practical Knowledge

traditional research has privileged knowledge that is produced by synthesizing data contributed by others, typically experts. Such a view keeps theory and practice separate. But knowledge can also be understood through the critical reflection of personal and collective experiences, whether these experiences are recent (Clare, 2006) or historical (fals Borda, 1987). in addition, knowledge can emerge as people do something in a critically reflective way. heron and Reason (1997) call these two other ways of knowing “experiential” and “prac-tical” respectively. valuing experiential and practical knowledge as-sumes that people can create new understanding that is grounded in their social involvements, which in turn creates a better informed practice that is guided by newfound insights (israel, Schulz, parker, & Becker, 1998).Evaluation of CCis can value such knowledge by trian-gulating the opinions of all stakeholders, encouraging them to critical-ly reflect on both their experience of, and their efforts within, the CCi.

provide Ongoing feedback

to ensure that the experiential and practical ways of ‘knowing’ de-scribed above are a collective exercise, research should include cycles of feedback and exchange among people (heron & Reason, 1997). Such co-operative inquiry implies that communities involved in re-


search own their knowledge (fals Borda, 1987). Community members are therefore in the best position to link what they collectively know to what they collectively do. Evaluation of CCis can incorporate regu-lar feedback of findings into the evaluation and intervention design, using multiple formats and forums that speak most clearly to respec-tive stakeholders (Chavez, duran, Baker, Avila, & Wallerstein, 2008).

facilitate democratic dialogue

following the critical theorist Jurgen habermas, Kemmis and mctag-gart (2005) call for researchers to create “communicative spaces.” Research participants are not seen as forced subjects of a research process or passive recipients of its findings. Rather, participants free-ly interact within research and with the knowledge it produces in or-der to reach inter-subjective agreement, mutual understanding, and unforced consensus. they are engaged in dialogue because they are encouraged to speak about what is important to them in ways that promote liberty, fraternity, and equality (Newbrough, 1995). Such a view implies that researchers are not only technicians, but also adopt the role of facilitator (Conde-frazier, 2006). Researchers become col-laborators rather than controllers, and research quality rests on the ability of researchers to stimulate, use, and interpret open discussion (heron & Reason, 1997). Such democratic dialogue provides the re-flective space in which theory and practice can be brought together. Evaluation of CCis can nurture this reflective space by incorporating methods that encourage interaction (e.g., focus groups and commu-nity forums), and by using feedback to facilitate further discussion of what existing findings mean for future action.

focus on a vision for the Common Good

A final facilitator for reflective practice is carrying out the research collaboration in such a way that people gain the collective capacity to imagine how the circumstances of their lives could be improved (Kemmis & mctaggart, 2005). indeed, the research process itself could be conceived of as a negotiation among stakeholders about the type of future they wish to co-create. if such a vision-focus is fostered within a research process, participants will gain licence to reflec-tively link their collective knowledge with their desired collective action. Within CCis there are often diverse stakeholder perspectives and interests, which are often in competition and at odds. the re-search process can act as a forum in which diverse perspectives and interests are aired and negotiated in a transparent and systematic way (Ochocka, moorlag, & Janzen, 2010).


thE WRiEN CASE ExAmplE

description of Waterloo Region immigrant Employment Network

the Waterloo Region immigrant Employment Network (WRiEN—pronounced “Ryan”) was launched in 2006 in Waterloo Region, a community of half a million in southern Ontario (see www.wrien.com). WRiEN was created to act as a local strategy to help ensure that the skills of immigrants would be more optimally used to the benefit of immigrants and their families, the region’s economy, and the community as a whole. it was developed in response to the well-documented under-utilization of immigrant skills across Canada (Al-boim, 2009; Wayland, 2006), including considerably higher rates of unemployment and under-employment, as well as lower earnings of immigrants in Waterloo Region relative to Canadian-born—this de-spite the higher education levels of immigrants (miedema & vande-belt, 2006). WRiEN was a comprehensive community initiative that provided a forum to explore, develop, and test innovative system change responses to immigrant employment (comprehensive). it did this through a broad-based and cross-sectoral engagement of com-munity stakeholders who identified their own solutions and priorities (community-building) (Janzen,Walton-Roberts, & Ochocka, in press).

WRiEN emerged as a result of extensive community consultation. led by the Centre for Community Based Research (CCBR), diverse community stakeholders were brought together to identify local needs, resources, and strategies in facilitating immigrants’ access to employment. Stakeholder groups included immigrants, business, local government, community organizations, education, and non-government funders, many of which had not been previously engaged on this issue. WRiEN was launched following an immigrant Skills Summit at which leaders from all stakeholder groups prioritized and committed to a comprehensive set of community action strate-gies (Janzen, hatzipantelis, & hogarth, 2005). WRiEN subsequently functioned as a body that convened and facilitated the collective ac-tion of these various local stakeholders in implementing the identi-fied strategies. WRiEN was among the first immigrant employment comprehensive community initiatives to be developed in a mid-size urban centre in Canada, if not beyond (mcfadden & Janzen, 2007).

Winning awards from both nonprofit and private sectors, WRiEN’s innovation lay in leveraging diverse community resources toward the common goal of immigrant employment, with few implementation models to draw on. WRiEN was hosted by the Greater Kitchener-


Waterloo Chamber of Commerce and consisted of a multi-stakeholder steering committee (privileging the immigrant and business perspec-tives), four active work groups (investments, employment initiatives, immigrant support, and qualification recognition and enhancement) and was supported by four staff. WRiEN’s structure was intended to be non-hierarchical with an emphasis on involving all community stakeholders. Core WRiEN funding was provided by local govern-ment and non-government funders. WRiEN was also successful in securing additional project funding to support work group priorities.

it is important to emphasize that WRiEN was not a direct service immigrant employment program. that is to say, WRiEN was not de-signed to directly match individual immigrants with employers (the region has numerous such programs). instead, it was a broad-based network designed to create the community conditions that would make successful immigrant employment more likely (i.e., systems change). it did this by facilitating diverse segments of the local com-munity to work together in new ways. for example, WRiEN emerged as a region-wide voice in advocating to senior levels of government on an array of immigrant employment issues. WRiEN was also in-strumental in launching and supporting a series of community initia-tives: an immigrant mentorship program, an immigrant internship program, an immigrant loan program (to support further education), an immigrant web portal, and an employer connections program. in all these examples WRiEN itself was not the lead organization but was the convener and catalyst in securing the necessary partners and resources to create and deliver “made-in-Waterloo” solutions (Wayland, 2007).

description of the WRiEN Evaluation

An ongoing process and outcome evaluation was built into WRiEN’s initial three-year mandate. Evaluation was intended to determine how and to what extent WRiEN’s multi-stakeholder and region-wide efforts were impacting the lives of immigrants, the local economy, and the broader community of Waterloo Region. the main research questions related to the structure of WRiEN, the implementation of its activities, outcomes observed to date, lessons learned, and recom-mendations for the future. A research team from the CCBR led the evaluation under the guidance of a cross-stakeholder steering group.

the evaluation used a mixed method design (tashakkori & teddlie, 2003) carried out in annual cycles over the three-year evaluation


period. methods that were repeated each year and that considered multiple stakeholder perspectives included key informant interviews (n = 50); focus group interviews (n = 52 in seven groups); participant observation of the WRiEN steering committee, work group meetings, and WRiEN-sponsored community events (total meetings/events = 25; almost 40% of all meetings/events); brief meeting tracking forms completed by participants at the end of each steering committee and work group meeting (n = 263); and an internal and external docu-ment review. in addition, an employers’ web survey (for private, pub-lic, and nonprofit sectors) was conducted in the first and last years of the study (n = 264).

the WRiEN evaluation adopted a participatory action research ap-proach that we defined as a “research approach that involves active participation of stakeholders, those whose lives are affected by the issue being studied, in all phases of research for the purpose of producing useful results to make positive changes” (Nelson et al., 1998, p. 9). three primary mechanisms were used to implement this approach. first, the CCBR evaluation team comprised members bringing content expertise (in immigrant employment and systems change evaluation), experiential knowledge (recent immigrants who had subsequently been unemployed or underemployed), and a com-mitment to advancing immigrant employment as an issue of so-cial justice. Second, a seven-member evaluation committee guided each step of the evaluation process. Committee members gave input into all evaluation tools, sampling criteria, participant recruitment, analysis, interpretation, and the dissemination of findings. mem-bers reflected WRiEN’s diverse stakeholder groups and had varying degrees of involvement within WRiEN. third, the evaluation was characterized by frequent feedback in multiple formats. in addition to the annual evaluation reports, feedback was provided via evalua-tion e-bulletins, summaries and presentations of individual methods, findings posted on the evaluation corner of the WRiEN website, and evaluation updates at all WRiEN steering committee meetings. in addition, a community forum was held each year at which time the year’s annual evaluation findings were reported.

the evaluation repeated the action reflection cycle (figure 1) three times (i.e., one cycle for each of the three years). following each year, the evaluation committee reconsidered how to adapt the evaluation process based on emergent learning. the research questions and methods remained relatively constant from year to year with minor changes. more pronounced changes were made in how the evaluation findings were shared from year to year, and how this subsequently


informed new practice within WRiEN (i.e., phase 4 of the action reflection cycle). in addition, the program’s theory of change was reviewed each year using an evolving logic model technique. the baseline and year 3 logic models are found in figures 2 and 3 respec-tively. these logic models served as reflective tools to consider what had been accomplished over the previous year, and looked forward to the emergent theory of what should be done in the upcoming year.

Evaluation findings highlighted the strength of WRiEN’s collabora-tive approach, but also pointed out limitations to date (dildar & Jan-zen, 2009; hogarth, Janzen, & Cushing, 2007; Janzen & dildar, 2008). WRiEN had increased community awareness about the issue of im-migrant employment and was successful in leveraging additional fi-nancial resources from senior levels of government. WRiEN was also successful in creating an increased level of trust among its partners that led to the formation of a series of new interventions dealing with specific aspects of immigrant employment (e.g., internship, mentor-ship, loan programs). however, further community engagement was needed, particularly among employers and recent immigrants, as was more attention on a rapid advocacy response and on marketing. Another significant constraint of WRiEN was its narrow focus on skilled professionals to the exclusion of unskilled workers.

liNKiNG WRiEN’S REflECtivE pRACtiCE tO thEORy

What did we learn about the role of evaluation in helping to shape and direct the evolving WRiEN intervention? Below we use the five facilitators of reflective practice that we previously identified as a frame to discuss our learnings. We also link our learnings to the evaluation literature, particularly that which has appropriated or adapted collaborative modes of inquiry. this is done to demonstrate how previous evaluation discourse has contained elements of partici-patory action research.

Be location-Based

the WRiEN initiative was situated in a community that had a long history of addressing immigrant employment issues. the CCBR and several service providers had provided nearly a decade of leadership on various research and community development projects prior to launching WRiEN (Centre for Community Based Research, 2010). those efforts included (a) research projects that analyzed the local employment and support needs and capacities of immigrants, (b) en-

72 The Canadian Journal of Program evaluaTion72Fi

gure

2W

RIE

N B

ase

Line

Log

ic M

odel

(200

6)

Wor

k G

roup

s (V

ertic

al)

Sam

ple

Prio

rity

Activ

ities

Mai

n C

ompo

nent

s

-Com

pile

and

di

strib

ute

info

rmat

ion

on

imm

igra

nt

empl

oym

ent

issu

es lo

cally

an

d na

tiona

lly

Info

rmat

ion

Cle

arin

g-ho

use

Stee

ring

Com

mitt

ee

(Hor

izon

tal)

Activ

ity

Area

s C

omm

uni-

catio

ns

Adv

ocac

y

Qua

lific

atio

n R

ecog

nitio

n/E

nhan

ce-

men

t

Imm

igra

nt

Sup

port

Imm

igra

nt

Attr

actio

n In

vest

men

ts

Net

wor

k C

oord

inat

ion

Com

mun

ity

Con

sulta

tion

Em

ploy

er

Initi

ativ

es

Dire

ct

Out

com

es

Ove

rarc

hing

O

utco

mes

Long

Ter

m

Visi

onar

y O

utco

mes

-Dev

elop

m

arke

ting

stra

tegy

for

entir

e ne

twor

k

-Con

duct

re

gion

-wid

e ad

voca

cy th

at

is in

form

ed b

y w

ork

grou

ps

and

com

mun

ity

cons

ulta

tions

-Det

erm

ine

linka

ges

and

enco

urag

e flo

w o

f in

form

atio

n ac

ross

the

netw

ork

-Pro

vide

fo

rum

for

com

mun

ity-

wid

e in

put i

nto

WR

IEN

pr

iorit

ies

-Ens

ure

finan

cial

vi

abili

ty o

f W

RIE

N

oper

atio

ns

-Attr

act $

from

se

nior

leve

ls

of g

ov’t

-Mar

ketin

g ca

mpa

ign

-Cas

e st

udie

s -E

mpl

oyer

re

sour

ce

guid

es

-Net

wor

king

op

portu

nitie

s

-Adv

ocat

e/

coor

dina

te

brid

ging

pr

ogra

ms

-Pro

mot

e ac

cess

to

finan

cial

re

sour

ces

-Co-

advo

cate

at

all

leve

ls

-Pro

vide

in

form

atio

n to

im

mig

rant

s -E

ncou

rage

or

gani

zatio

nal

impr

ovem

ent

Bra

nd/m

arke

t W

ater

loo

to

imm

igra

nts

-D

eplo

y w

ebsi

te

-Gat

her

data

em

ploy

men

t re

sour

ces

Mor

e st

rate

gic

recr

uitm

ent

and

reta

inm

ent o

f im

mig

rant

sk

ills

Bet

ter e

quip

ping

of

imm

igra

nts

in

over

com

ing

syst

emic

em

ploy

men

t ba

rrier

s

Incr

ease

d re

sour

ces

leve

red

and

mor

e ef

ficie

nt

allo

catio

ns

Mor

e st

rate

gic

orie

ntat

ion

and

supp

ort o

f em

ploy

ers

in

hirin

g im

mig

rant

s an

d su

cces

sful

ly

inte

grat

ing

them

in

to th

e w

orkp

lace

Bet

ter

reco

gniti

on a

nd

enha

ncem

ent

of th

e pr

ior

lear

ning

and

cr

eden

tials

of

imm

igra

nts

Impr

oved

co

llabo

ratio

n,

partn

ersh

ips

and

syne

rgiz

es

amon

g st

akeh

olde

rs

Impr

oved

regi

on-

wid

e sy

stem

ic

advo

cacy

and

be

tter l

obby

ing

of

seni

or

gove

rnm

ents

re:

po

licy

chan

ges

Incr

ease

d aw

aren

ess

of

imm

igra

nt

empl

oym

ent

issu

es a

nd

impr

oved

acc

ess

to re

leva

nt in

fo

Mor

e im

mig

rant

-fri

endl

y w

orkp

lace

s

Mor

e in

clus

ive

hirin

g pr

actic

es/s

trate

gies

am

ong

empl

oyer

s an

d re

vers

ed tr

end

of

imm

igra

nt e

mpl

oym

ent

barri

ers

Incr

ease

d lik

elih

ood

of

empl

oyer

s w

ith v

acan

cies

hi

ring

imm

igra

nts/

im

mig

rant

s ar

e m

ore

visi

ble

and

succ

essf

ul w

ithin

re

crui

tmen

t and

sel

ectio

n pr

oces

ses

Impr

oved

sen

se

of b

elon

ging

w

ithin

Wat

erlo

o R

egio

n by

im

mig

rant

s, a

nd

imm

igra

nts

mor

e lik

ely

to s

tay

Mor

e an

d m

ore

effe

ctiv

e im

mig

rant

su

ppor

ts, w

ith

decr

ease

d ga

ps/o

verla

ps

Gre

ater

coh

eren

cy

and

effe

ctiv

enes

s of

re

gion

al/p

rovi

ncia

l/ fe

dera

l im

mig

rant

em

ploy

men

t pol

icy

Incr

ease

d #

of

empl

oyer

s an

d go

vern

men

t/ co

mm

unity

age

ncie

s se

tting

ann

ual

prog

ram

pla

ns a

nd

allo

tting

fund

s In

crea

sed

num

ber o

f im

mig

rant

s an

d em

ploy

ers

acce

ssin

g av

aila

ble

supp

orts

Incr

ease

d m

arke

tabi

lity

of

imm

igra

nt s

kills

Bet

ter

prio

ritiz

atio

n an

d be

tter

impl

emen

tatio

n of

a

com

preh

ensi

ve,

regi

on-w

ide

plan

Em

ploy

ers

mor

e aw

are

of

imm

igra

nt

empl

oym

ent

issu

es

Impr

oved

at

tract

ion

of

imm

igra

nt ta

lent

re

lativ

e to

labo

ur

mar

ket t

rend

s

Impr

oved

im

mig

rant

wel

l-be

ing

Stro

ng L

ives

…

Stro

ng E

cono

my…

St

rong

Com

mun

ities

…

Incr

ease

d re

gion

al

econ

omic

pr

ospe

rity

Hea

lthie

r, m

ore

vibr

ant a

nd m

ore

incl

usiv

e co

mm

uniti

es


Figu

re 3

WR

IEN

Yea

r 3 L

ogic

Mod

el (2

009)

Prio

rity

Activ

ities

Mai

n C

ompo

nent

s

Activ

ity

Area

s

Foun

datio

nal

Cha

nge

Tang

ible

C

omm

unity

C

hang

e

Long

Ter

m

Cha

nge

Wor

k G

roup

s (W

ithin

issu

e)

-Com

pile

and

di

strib

ute

imm

igra

nt

empl

oym

ent

libra

ry

-Reg

ular

em

ail

upda

tes

-Dis

tribu

te

“Tod

ay’s

W

orkf

orce

” e-

bulle

tin

-Web

de

velo

pmen

t

Info

rmat

ion

Cle

arin

g-ho

use

Ste

erin

g C

omm

ittee

(C

ross

Net

wor

k) C

omm

uni-

catio

ns a

nd

Mar

ketin

g A

dvoc

acy

Qua

lific

atio

n R

ecog

nitio

n/E

nhan

ce-m

ent

Imm

igra

nt

Supp

ort

Inve

stm

ents

C

omm

unity

Pl

anni

ng

Com

mun

ity

Con

sulta

tion

and

Enga

gem

ent

Empl

oym

ent

Initi

ativ

es

-Dev

elop

a

com

preh

ensi

ve

mar

ketin

g /

com

mun

icat

ion

stra

tegy

for e

ntire

ne

twor

k

-Con

duct

regi

on-

wid

e ad

voca

cy

that

is in

form

ed

by w

ork

grou

ps

and

com

mun

ity

cons

ulta

tions

-D

evel

op a

loca

l ra

pid

resp

onse

ad

voca

cy

stra

tegy

-Det

erm

ine

linka

ges

and

enco

urag

e flo

w o

f in

form

atio

n ac

ross

the

netw

ork

- Ini

tiate

and

su

ppor

t new

im

mig

rant

em

ploy

men

t in

terv

entio

ns

-Org

aniz

e pu

blic

fo

rum

for i

nput

an

d in

volv

emen

t w

ithin

WR

IEN

pr

iorit

ies

- Stri

ke a

n im

mig

rant

ou

treac

h ta

sk

forc

e to

invo

lve

imm

igra

nts

-Ens

ure

finan

cial

vi

abili

ty o

f WR

IEN

op

erat

ions

-A

ttrac

t $ fr

om

seni

or le

vels

of

gov’

t - S

treng

then

the

role

of t

he g

roup

to

util

ize

non-

finan

cial

re

sour

ces

-Impl

emen

t and

ex

pand

em

ploy

er

enga

gem

ent

stra

tegy

-P

rovi

de

netw

orki

ng

oppo

rtuni

ties

for

empl

oyer

s an

d im

mig

rant

s

-Adv

ocat

e fo

r cr

eden

tial

asse

ssm

ent

-Lau

nch

Imm

igra

nt L

oan

Prog

ram

-A

dvoc

ate

for

Fede

ral p

oint

s sy

stem

- A

dvoc

ate

for

mor

e br

idgi

ng

prog

ram

s

-Co-

advo

cate

at

all l

evel

s -P

rovi

de

info

rmat

ion

to

imm

igra

nts

-Sup

port

prog

ram

s fo

r Im

mig

rant

in

tegr

atio

n in

the

com

mun

ity a

nd

the

wor

kpla

ce

Few

er s

yste

mic

bar

riers

face

d by

im

mig

rant

s in

find

ing

and

mai

ntai

ning

mea

ning

ful e

mpl

oym

ent

Incr

ease

d nu

mbe

r of

imm

igra

nts

and

empl

oyer

s ac

cess

ing

avai

labl

e su

ppor

ts

Incr

ease

d m

arke

tabi

lity

of

imm

igra

nt s

kills

Impr

oved

im

mig

rant

wel

l-be

ing

Stro

ng L

ives

…

Stro

ng E

cono

my…

S

trong

Com

mun

ities

…

Incr

ease

d re

gion

al

econ

omic

pr

ospe

rity

Hea

lthie

r, m

ore

vibr

ant a

nd m

ore

incl

usiv

e co

mm

uniti

es

Incr

ease

d re

sour

ces

leve

rage

d an

d m

ore

effic

ient

allo

catio

ns

Mor

e re

spon

sive

regi

on-w

ide

advo

cacy

to e

mer

ging

loca

l iss

ues

and

bette

r lob

byin

g of

sen

ior

gove

rnm

ents

re: p

olic

y ch

ange

s

Incr

ease

d #

of e

mpl

oyer

s an

d go

vern

men

t/ co

mm

unity

age

ncie

s se

tting

an

nual

pro

gram

pla

ns a

nd

allo

tting

fund

s

Incr

ease

d aw

aren

ess

of im

mig

rant

em

ploy

men

t is

sues

and

impr

oved

acc

ess

to re

leva

nt

info

rmat

ion

(incl

udin

g su

cces

s st

orie

s)

Bet

ter p

riorit

izat

ion,

com

mun

icat

ion

and

publ

icity

of a

com

preh

ensi

ve, r

egio

n-w

ide

plan

Incr

ease

d co

mm

unity

rea

dine

ss fo

r cha

nge:

Pre

-WR

IEN

act

iviti

es c

ause

d pe

ople

see

the

pote

ntia

l of c

olla

bora

tion

and

mot

ivat

ed p

eopl

e to

w

ant t

o co

ntrib

ute

to th

is u

niqu

e co

mm

unity

opp

ortu

nity

(en

ergi

zed

visi

on).

Incr

ease

d st

akeh

olde

r (pa

rticu

larly

em

ploy

er

and

imm

igra

nt) e

ngag

emen

t Im

prov

ed e

nviro

nmen

t for

col

labo

ratio

n,

partn

ersh

ips

and

syne

rgiz

es a

mon

g st

akeh

olde

rs

(sha

red

owne

rshi

p)

Info

rm a

nd m

ake

requ

ests

of

each

oth

er

Dec

reas

ed g

aps

and

over

laps

in

loca

l im

mig

rant

sup

port

prog

ram

s (n

ew c

olla

bora

tive

inte

rven

tions

)

Impr

oved

sen

se o

f bel

ongi

ng w

ithin

W

ater

loo

Reg

ion

by im

mig

rant

s,

and

imm

igra

nts

mor

e lik

ely

to s

tay

Mor

e im

mig

rant

-frie

ndly

wor

kpla

ces

with

suc

cess

ful e

mpl

oym

ent

outc

omes

Incr

ease

d op

portu

nitie

s fo

r im

mig

rant

s to

in

tera

ct w

ith b

usin

ess

com

mun

ity

Incr

ease

d ac

cess

to

educ

atio

nal/s

kill

upgr

adin

g fo

r im

mig

rant

s

Mor

e in

clus

ive

hirin

g pr

actic

es

amon

g em

ploy

ers

and

incr

ease

d tre

nd o

f im

mig

rant

s fin

ding

wor

k in

thei

r fie

ld

Gre

ater

coh

eren

cy a

nd

effe

ctiv

enes

s of

re

gion

al/p

rovi

ncia

l/fed

eral

im

mig

rant

em

ploy

men

t pol

icy

Mor

e su

ppor

t for

em

ploy

ers

to h

ire

imm

igra

nts

and

succ

essf

ully

in

tegr

ate

them

into

the

wor

kpla

ce

(lang

uage

& a

ccul

tura

tion

supp

orts

)

Bet

ter r

ecog

nitio

n an

d en

hanc

emen

t of t

he p

rior

lear

ning

and

cre

dent

ials

of

imm

igra

nts


gagement of diverse stakeholders in gathering ideas and sugges-tions for action, (c) monitoring of relevant initiatives in Ontario and beyond, and (d) extensive work on building a community of activists around immigrant employment issues. these initiatives helped build a foundation of local knowledge, a strong sense of local community need, and an engaged local stakeholder base. But, perhaps most im-portantly, these initiatives also laid the groundwork to identify the priorities for local action that WRiEN could champion.

the WRiEN evaluation built on this local momentum. to begin, in-volving the CCBR to head evaluation efforts ensured that the evalu-ation process would be carried out with a deep understanding of local context and history. Evaluations of system change initiatives such as WRiEN require that a clear bounding of the system is established where social problems, the multiple levels of the system, and the set-tings, organizations, and actors are clearly understood and defined (foster-fishman et al., 2007; Stake, 1978). in addition, the CCBR had become a trusted entity in bringing new stakeholders together, and could now leverage that trust toward cross-stakeholder participation within the evaluation. A lack of trust has been identified as a leading challenge in promoting successful community research partnerships (Becker, israel, & Allen, 2005), including between evaluators and community members (Nelson, Janzen, Ochocka, & trainor, 2010). the WRiEN evaluation benefited from its past history of trust as evaluators facilitated stakeholders to conceptualize and agree upon their unique local theory of change. in this way the evaluation team became an active member of the intervention team (patton, 2011). in the words of one WRiEN staff, “the evaluation was not leading, but helping the leaders to lead.”

value Experiential and practical Knowledge

CCis such as WRiEN are typically conceived by diverse stakehold-ers and grounded in their lived experiences and perspectives. the program theory, goals, and objectives are driven and negotiated by stakeholders (Kubisch et al., 1998). this reflected the efforts lead-ing up to WRiEN and its subsequent implementation. the WRiEN evaluation built on this tradition by inviting stakeholders to (a) rec-reate objectives and actions in each of the stages of the intervention development, (b) critically reflect on the WRiEN’s contributions and achievements from year to year, and (c) develop solutions for improve-ments and options for meeting the overall goal of the intervention. the WRiEN evaluation therefore mirrored patton’s (2008) “theory of


action framework” by encouraging practitioners to examine, reflect on, and deal with the discrepancies between their “espoused theory” (how practitioners explain what they are attempting to do) and their “theory in use” (what their behaviour reveals about what actually guides what they do).

valuing the experiential and practical knowledge of evaluation par-ticipants is a key component of reflective practice (Nelson, Ochocka, Janzen, trainor, & lauzon, 2004). Within WRiEN this was concretely achieved through three means. first, many of the questions asked within the evaluation encouraged participants to reflect both on their experience of being involved in WRiEN (e.g., “to what extent was your opinion respected within this WRiEN meeting?”), as well as what they practically accomplished through the involvement (e.g., “to what extent did this meeting move ahead the agenda of this work group/steering committee?”). in addition, the design of the evalua-tion emphasized triangulation of these opinions through multiple methods (e.g., surveys, key informant interviews, observations, focus group, meeting tracking) and from multiple stakeholder perspectives (immigrants, business, government, service providers, education, non-governmental funders). finally, the process of analysis encour-aged collective critical reflection on emerging evaluation themes. data from the multiple mixed-methods were initially analyzed in parallel by the research team. preliminary evaluation findings were then verified and shaped through discussions among WRiEN partici-pants who sat on the evaluation and steering committees.

theory that is grounded in social experiences can lead to better in-formed and more effective practice (Cargo & mercer, 2008; israel et al., 1998). the WRiEN evaluation validated the experiences (whether positive or negative) of the various stakeholders involved in WRiEN, and leveraged that validation in moving participants from a place of opinion and critique toward constructive new action. that is not to say that valuing experiential and practical evidence was without challenges. for example, expectations were raised among evaluation participants that their opinions actually mattered in reshaping the future of the intervention.

the stated openness to forgo a typical fixed theory of change in favour of “learning as we go” (Nelson et al., 1998) certainly intro-duced the challenge of raised expectations that WRiEN would be responsive to evaluation findings. While the goals, objectives, and actions of WRiEN were initially agreed upon, the evaluation was a built-in mechanism to ensure that the numerous and diverse objec-


tives would be continuously negotiated over time, based on the ac-knowledgement and documentation of the experiential knowledge. in this way, the evaluation promoted an organizational learning system (Cousins, Goh, Clark, & lee, 2004). to illustrate, year 3 evaluation findings obtained from multiple stakeholders’ experiences revealed a need for much stronger promotion, communication, and marketing of WRiEN as one of the main priorities for action even though this was not a prominent part of the original set of activities. the challenge was then to deliver on this new urgent set of actions.

provide Ongoing feedback

Knowledge utilization within evaluation is maximized when efforts are made to share evaluation findings in clear, useful, and respectful language targeted at multiple stakeholder perspectives (Nelson et al., 2010; patton, 2008). A number of feedback strategies were used in the WRiEN evaluation to maximize reflective practice. the first dealt with the timeliness of feedback. Whether giving feedback of indi-vidual evaluation methods or providing an annual overall summary, an emphasis was placed on providing feedback as quickly as possible. Closing the gap between data collection and analysis was important in order for those involved in WRiEN (i.e., staff, work groups, steer-ing committee) to make evidence-based decisions that were informed by current information. “Real-time evaluation” was how one member of the business community consequently dubbed the evaluation. the timeliness of generating feedback was essential, particularly in mo-ments when dissatisfaction was detected and when new solutions were emerging. Such a situation occurred between years 2 and 3, when the evaluation results provided grounds for action that led to significant reshaping of its original ambitious goals and priorities.

A second strategy was to use multiple stages and forms of feedback. Regular feedback of evaluation processes and preliminary findings was provided to evaluation committee members in e-news format and during meetings. Summary bulletins for selected methods (e.g., employer web survey and meeting tracking forms) were also distrib-uted and presented to both the evaluation and steering committees for verification and to begin the discussion of implications for future action. At year end, the final report went through multiple stages of feedback. it began with the evaluation committee verifying main themes, providing interpretive comments about how best to frame these themes, and identifying options for action (i.e., recommenda-tions). A presentation to the WRiEN steering committee again served


to verify and frame themes, but, more importantly, to stimulate decision-making about how evaluation recommendations could be translated into concrete action steps. finally, the year-end evalua-tion findings were presented to the broader community via a report posted on the WRiEN website and at an open forum.

this leads to the third feedback strategy. Evaluation can be seen as a servant in support of a broader social change movement, particularly for those with limited power and opportunity (mertens, 2009; Nelson et al., 2004). in this light, the feedback loops within WRiEN’s evalu-ation findings could be viewed as an opportunity to constructively advance an immigrant employment agenda. to this end, WRiEN findings were carefully positioned to provide constructive rather than destructive criticism, pointing to the value of learning and evo-lution rather than simply examining and judging what worked and what did not. Such a tack enabled participants to provide not only reflections and comments but also options and scenarios for improv-ing the advancement of immigrant employment.

in summary, a commitment to utilization was foregrounded with researchers adopting feedback formats that tried to connect with di-verse stakeholder audiences (patton, 2008). providing such feedback served to engage stakeholders within the WRiEN intervention: (a) it helped stakeholders continuously reflect and stay on track with their own goals, (b) it provided a venue for timely suggestions for improvements, and (c) it served as a tool for synthesizing knowledge and translating it into desired actions. in other words, evaluation feedback was used to embed continuous reflection into the “WRiEN culture” (Cousins et al., 2004).

facilitate democratic dialogue

in collaborative approaches to evaluations the evaluator may act as a facilitator and co-agent of change (Janzen, 2011; Nelson et al., 2010). the evaluator’s role is not only to enable the generation and synthesizing of new knowledge, but also to lead participants into a democratic dialogue that helps them to co-create knowledge and use it as a catalyst for future action. the facilitation of democratic dialogue resonates with the concept of “process use” that foregrounds the learning that occurs during the evaluation process itself (patton, 2008). learning, including through evaluation, is viewed as a social process (Amo & Cousins, 2007). therefore, how an evaluation is con-ducted (and not simply its findings) may lead evaluation stakehold-


ers to acquire new knowledge and to learn new behaviours and skills (Alkin & taut, 2003).

Within CCis, facilitating democratic dialogue is encouraged by an openness to an evolving program theory that is ever responsive in the emergent local setting (lafferty & mahoney, 2003). for example, WRiEN’s first-year evaluation revealed that despite considerable preparatory work, WRiEN’s role as a systems change intervention was still not understood across participants. Without tangible and visible accomplishments in the first year, some community members began to question the intervention, seeing it as being too broad and removed from their day-to-day realities. the steering committee dis-cussions about these preliminary evaluation findings, however, led to the reframing of findings by recognizing the importance of “founda-tional” change (e.g., increased awareness of immigrant employment issues and improved environment for synergistic partnerships) as precursors to more tangible outcomes (e.g., new collaborative inter-ventions, improved hiring practices). A bamboo analogy was used to explain that considerable root development is needed (foundational change) prior to bamboo shoots breaking through the earth (tangible change). Rather than being simply frustrated with the slow pace of tangible change, stakeholders came to articulate a local logic of how observed foundational changes could lead to desired longer-term impacts.

As the above example illustrates, democratic dialogue was stimu-lated within the WRiEN evaluation primarily in two phases of the reflective action cycle: information gathering (phase 2) and informa-tion sharing (phase 4). With regards to information gathering, a mix of qualitative and quantitative as well as individual and group methods provided variety in dialogue formats. Similarly, the multiple feedback sessions in phase 4 allowed multiple opportunities for ex-change of opinion. in both cases a range of stakeholder perspectives was included, but always emphasizing the voice of WRiEN’s two central stakeholder groups (immigrants and employers). following the tradition of participatory evaluation, evaluation utilization was therefore maximized by actively engaging these multiple stakehold-ers throughout the research process, recognizing that their per-spectives were critical given that knowledge is socially constructed (Cousins & Earl, 1992).

facilitating democratic dialogue was particularly important when transformational changes were desired (such as in year 2 of WRiEN’s operation). At this time the stakeholders were clearly prepared to


look at whether the intervention was reaching its main goals and were much clearer in their judgement of successes and challenges. during this stage, the researchers took additional steps to clarify the importance of carefully interpreting results and exploring proactive solutions. this process ensured that the participants felt heard while their issues were addressed in a constructive and mutually respect-ful fashion. moreover, those involved in the interpretation of findings and the generation of recommendations (i.e., evaluation and steering committees) felt empowered to offer solutions that led to improve-ments and desired changes (fetterman & Wandersman, 2005).

facilitating democratic dialogue was not without its challenges. to begin, facilitating democratic facilitation can be resource-intensive. WRiEN staff in particular acknowledged the strain that additional information gathering placed on their busy workloads. Consistent with the experience of staff within other collaborative system change evaluations (e.g., Janzen, Nelson, hausfather, & Ochocka, 2007), WRiEN staff saw the direct benefit of evaluation while also not-ing resource challenges in supporting its activities. Second was the challenge of engaging the full range of stakeholders in dialogue. As with other efforts to engage employers on immigrant matters (public policy forum, 2004), the business sector in particular proved most challenging to engage. Consequently, in the first year of the evaluation few employers were included in the key informant method (contrary to the sampling frame). however, participation in dialogue increased over time as the business sector became more involved in WRiEN and its evaluation and offered its unique insights into im-migrant hiring and retention.

focus on a vision for the Common Good

it is not uncommon for community coalitions to be composed of partners with disparate, if overlapping, agendas, expectations, and interests (Baum, 2001). Such was the case with WRiEN. for example, partners came into the intervention with differing histories—some having grappled with the issue of immigrant employment for many years if not decades, while others being relatively new to the is-sue. Some partners were ready for a systems change intervention (recognizing the limitations of individual-level efforts in addressing systemic barriers to immigrant employment), while others were en-trenched in a mindset of the primacy of individual-level efforts and outcomes. in addition, some partners were primarily motivated in joining WRiEN as a means of promoting social justice: working to


give immigrants equitable opportunity to participate in the labour market and consequently in community life. Others were primarily attracted to an economic imperative agenda, recognizing that an aging population meant that the local (as well as national) economy was entering a time of labour skill shortages that immigrants poten-tially could fill. finally, partners held significantly different social norms for pursuing healthy partnerships. Even the seemingly trivial expectation of what constituted a fruitful meeting exposed deep dif-ferences among partners (i.e., the brief early morning meetings with rational decision-making and a bias for action expected by the busi-ness community, contrasted with the constant connection to lived experience and ample discussion to understand the depth of issues expected by immigrant service providers).

it was within this context that the evaluation was able to help focus the disparate partnership on a common vision. the evaluation acted as a forum in which these diverse perspectives and interests were aired and negotiated in a transparent and systematic way. All stake-holder perspectives were sought as evaluation data. All stakeholders were also invited to re-examine and collectively recommit to their collective vision within the various multi-stakeholder feedback and discussion formats. Still, keeping the initial common vision alive across many stakeholder groups was difficult.

What proved helpful in this regard was to continually focus stake-holders, despite their different motivations, on what they considered to be the “common good.” in evaluation language this translates to reaching a shared understanding of desired outcomes. the slo-gan that stakeholders had crafted for the original immigrant Skills Summit (“strong lives, strong economy, strong community”) was subsequently reframed as long-term goals that remained constant throughout the evaluation (see bottom of the logic models in figures 2 and 3). Such a three-pronged “common good” satisfied both social justice and economic imperative agendas, while framing them as interdependent outcomes within WRiEN’s “official” program theo-ry. Consequently, these long-term outcomes served as a launching point for the ongoing discussion of what intermediary outcomes were expected and what priority activities were needed to reach them. Stakeholders could therefore see how their specific efforts fit into the broader (and shared) system change goals.

Using evaluations within system change interventions as a platform to bring clarity around theory of change is certainly not unique to


WRiEN (e.g., foster-fishman & long, 2009; Janzen et al., 2007; Ku-bisch et al., 1998). And using evaluation data as a means of facilitat-ing partnership commitment and activities is also well documented (e.g., Natasi & hitchcock, 2009). the important point to make here is that evaluation that focuses on a vision of the common good offers participants an opportunity to reflectively link their collective knowl-edge with their desired collective action. in other words, evaluation can build a new sense of community among previously separated stakeholder interests—a community that inspires people to work together toward a common goal (Stringer, 2007).

CONClUSiON

in this article we discussed how evaluation can help shape and direct CCis. this was done by offering the Waterloo Region immigrant Em-ployment Network (WRiEN) as a case example to illustrate reflec-tive practice—how theory and practice can be intertwined through evaluation. By framing our reflections around five facilitators that interconnect theory and practice, and by referring to literature on collaborative evaluation, we have placed our discussion within a broader body of knowledge. this body of knowledge represents two parallel and complementary knowledge bases; on the one hand what is known about reflective practice through multidisciplinary writ-ings on participatory action research, and on the other hand what is known about reflective practice through collaborative inquiry within the trans-discipline of evaluation. Although obviously not exhaus-tive, our intent was to bridge the gap that often exists between these two knowledge bases.

the WRiEN case study outlined in this article contributes to the emerging discourse distinguishing formative from developmental evaluation. proponents of developmental evaluation (e.g., Gamble, 2008; patton, 2008, 2011) emphasize that developmental evaluation develops a newly emerged intervention model, while formative evalu-ation improves an existing intervention model. intervention models need to be developed (rather than tested and tweaked) when their contexts are complex with low stakeholder agreement and certainty about outcomes. We agree that this distinction is an important one, and view the WRiEN example as being clearly developmental. the emphasis on emerging program theory within WRiEN explicitly in-vited ongoing reflective practice and allowed evaluators to position themselves as part of what patton calls the “adaptive management team.” Although WRiEN had developed a baseline program theory,


this theory was significantly renegotiated over time. the WRiEN experience therefore suggests that perhaps it is less a matter of whether an intervention model does or does not pre-exist (can an intervention ever be entirely theory-free?) than it is a matter of how loosely or tightly a model is held. from this angle the formative-de-velopmental distinction could be conceived of as a continuum rather than as two discrete categories.

there is a second contribution to the discussion about developmen-tal evaluation. in this article we reflected on WRiEN’s evaluation through five facilitators of reflective practice that were gleaned from past literature. While in his book patton (2011) introduces develop-mental evaluation as a new approach to evaluation (as distinct from formative and summative evaluation), the WRiEN case example demonstrates that using research as a means of facilitating ongo-ing reflective practice within complex environments is neither new nor novel. Although the term “developmental evaluation” was not yet coined in the literature at the beginning of WRiEN’s evaluation, the evaluation can be seen to stand on the shoulders of a deep par-ticipatory action research tradition that spans more than 60 years. it also draws on the growing interest among evaluators pursuing collaborative forms of inquiry that adopt constructivist and other postmodern paradigms of research. these paradigms of research are comfortable with moving beyond the objectivity, linear cause-and-effect assumptions, and controlled environments of positivistic science, particularly when evaluating innovation (Janzen & Wiebe, 2010; Nelson et al., 2010). patton’s developmental evaluation does add helpful connections to complexity theory and the emerging body of knowledge related to social innovation. it may yet prove to become a consensus label useful in articulating and promoting a postmodern approach to evaluation. time will tell. Still, regardless of which label we use, we believe that CCis provide fertile ground to further explore the link between evaluation, reflective practice, and interventions that deal with complexity.

ACKNOWlEdGEmENtS

the authors would like to acknowledge the WRiEN evaluation and steering committees for their input and support during the evalua-tion process, and drs. Geoffrey Nelson and terry mitchell for com-ments on previous section drafts.


REfERENCES

Alboim, N. (2009). Adjusting the balance: Fixing Canada’s economic immi-gration policies. toronto, ON: maytree foundation.

Alkin, m. C., & taut, S. m. (2003). Unbundling evaluation use. Studies in Educational Evaluation, 29, 1–19.

Amo, C., & Cousins, J. B. (2007). Going through the process: An examina-tion of the operationalization of process use in empirical research on evaluation. New Directions for Evaluation, 116, 5–26.

Baker, t. E. (2003). Evaluating community collaborations: An overview. in t. E. Baker (Ed.), Evaluating community collaborations (pp. 1–18). New york, Ny: Springer.

Baum, h. S. (2001). how should we evaluate community initiatives? Jour-nal of the American Planning Association, 67(2), 147–158.

Becker, A. J., israel, B. A., & Allen, A. J. (2005). Strategies and techniques for effective group processes in CBpR partnerships. in B. A. israel, E. Eng, A. J. Schulz, & E. A. parker (Eds.), Methods in community-based participatory research for health (pp. 52–72). San francisco, CA: Jossey-Bass.

Cargo, m., & mercer, S. (2008).the value and challenges of participatory re-search: Strengthening its practice. Annual Review of Public Health, 29, 325–350.

Centre for Community Based Research. (2010). Mobilizing Waterloo Region around immigrant employment. Retrieved from www. communitybasedresearch.ca/page/view/immigrant_Employment.html

Chavez, v., duran, B., Baker, Q. E., Avila, m. m., & Wallerstein, N. (2008). the dance of race and privilege in community based participatory research. in m. minkler & N. Wallerstein (Eds.), Community-based participatory research for health (pp. 91–106). San francisco, CA: Wiley.

Chen, h.-t. (2005). Practical program evaluation: Assessing and improv-ing planning, implementation, and effectiveness. Newbury park, CA: Sage.

Clare, R. C. (2006). putting faith into action: A model for the North Ameri-can middle class. Religious Education, 101(3), 368–389.


Conde-frazier, E. (2006). participatory action research: practical theology for social justice. Religious Education, 101, 321–329.

Cousins, J. B., & Earl, l. m. (1992). the case for participatory evaluation. Educational Evaluation and Policy Analysis, 14(4), 397–418.

Cousins, J. B., Goh, S. C., Clark, S., & lee, l. l. (2004). integrating evalua-tive inquiry into the organizational culture: A review and synthesis of the knowledge base. Canadian Journal of Program Evaluation, 19(2), 99–141.

dildar, y., & Janzen, R. (2009). Waterloo Region Immigrant Employment Network (WRIEN) final evaluation report. Kitchener, ON: Centre for Community Based Research.

fals Borda, O. (1987). the application of participatory action-research in latin America. International Sociology, 2(4), 329–347.

fetterman, d. m., & Wandersman, A. (2005). Empowerment evaluation principles in practice. New york, Ny: Guilford.

foster-fishman, p. G., & long, R. (2009). the challenges of place, capacity and systems change: the story of yes We Can. Foundation Review, 1(1), 69–84.

foster-fishman, p. G., Nowell, B., & yang, h. (2007). putting the system back into system change: A framework for understanding and chang-ing organizational and community systems. American Journal of Community Psychology, 39(3), 197–215.

freire, p. (1970). Pedagogy of the oppressed. New york, Ny: Seabury press.

Gamble, J. A. A. (2008). A developmental evaluation primer. montreal, QC: J.W. mcConnell family foundation.

heron, J., & Reason, p. (1997). A participatory inquiry paradigm. Qualita-tive Inquiry, 3(3), 274–294.

hogarth, K., Janzen, R., & Cushing, S. (2007). Year one evaluation report for Waterloo Region Immigrant Employment Network (WRIEN). Kitch-ener, ON: Centre for Community Based Research.

israel, B., Eng, E., Schulz, A. J., & parker, E. A. (2005). introduction to methods in community-based participatory research for health. in B. A. israel, E. Eng, A. J. Schulz, & E. A. parker (Eds.), Methods in


community-based participatory research for health (pp. 3–26). San francisco, CA: Jossey-Bass.

israel, B., Schulz, A., parker, E., & Becker, A. (1998). Review of community-based research: Assessing partnership approaches to improve public health. Public Health, 19, 173–202.

Janzen, R. (2011). Reaching out to multicultural neighbours: Stories that evaluate and encourage innovative church outreach. Unpublished dissertation. Wilfrid laurier University, Waterloo, ON.

Janzen, R., & dildar, y. (2008). Year two evaluation report for the Waterloo Region Immigrant Employment Network (WRIEN). Kitchener, ON: Centre for Community Based Research.

Janzen, R., hatzipantelis, m., & hogarth, K. (2005). Immigrant skills sum-mit: Waterloo Region conference proceedings. Kitchener, ON: Centre for Research and Education in human Services.

Janzen, R., hatzipantelis, m., vinograd, J., & Kellerman, m. (2004). Good practice and resource guide: Community needs assessments and serv-ice evaluations in Military Family Resource Centres. Kitchener, ON: Centre for Community Based Research (formerly Centre for Re-search and Education in human Services).

Janzen, R., Nelson, G., hausfather, N., & Ochocka, J. (2007). Capturing system level activities and impacts of consumer-run organizations: methods that inform future action. American Journal of Community Psychology, 39(Special issue on systems change evaluation), 287–299.

Janzen, R., Walton-Roberts, m., & Ochocka, J. (in press). immigrant integra-tion and inclusion in Waterloo Region. in J. Biles & C. Andrew (Eds.), Immigration, integration and inclusion in Ontario cities. montreal, QC: mcGill-Queen’s University press.

Janzen, R., & Wiebe, d. (2010). putting God in the logic model: developing a national framework for the evaluation of faith-based organizations. Canadian Journal of Program Evaluation, 25(1), 1–26.

Kemmis, S., & mctaggart, R. (2005). participatory action research. in N. K. denzen & y. S. lincoln (Eds.), Handbook of qualitative research (3rd ed.; pp. 559–603). thousand Oaks, CA: Sage.

Kubisch, A. C., fulbright-Anderson, K., & Connell, J. p. (1998). Evalu-ating comprehensive community initiatives: A progress report. in


K. fulbright-Anderson, A. C. Kubisch, & J. p. Connell (Eds.), New approaches to evaluating community initiatives: Volume 2–Theory, measurement, and analysis (pp. 1–14). Washington, dC: Aspen in-stitute.

lafferty, C. K., & mahoney, C. A. (2003). A framework for evaluating com-prehensive community initiatives. Health Promotion Practice, 4(31), 31–44.

lewin, K. (1948). Action research and minority problems. in G. W. lewin (Ed.), Resolving social conflicts (pp. 201–216). New york, Ny: harper & Row.

mackenzie, m., & Blamey, A. (2005). the practice and the theory: lessons from the application of a theory of change approach. International Journal of Theory, Research and Practice, 11(2), 151–168.

mcfadden, p., & Janzen, R. (2007). the importance of immigrants to Water-loo Region’s prosperity: A dynamic collaborative community process. Our Diverse Cities, 4(fall), 104–107. Retrieved from http://canada.metropolis.net/publications/index_e.htm

mclaughlin, J. A., & Jordan, G. B. (2004). Using logic models. in J. S. Wholey, h. p. hatry, & K. E. Newcomer (Eds.), Handbook of practical program evaluation (pp. 7–32). thousand Oaks, CA: Sage.

mertens, d. m. (2009). Transformative research and evaluation. New york, Ny: Guilford press.

miedema, J. m., & vandebelt, d. (2006). Immigrants and growth: A look at health and employment in Waterloo Region. Waterloo, ON: Region of Waterloo public health.

minkler, m., & Wallerstein, N. (2008). introduction to community based participatory research. in m. minkler & N. Wallerstein (Eds.), Com-munity-based participatory research for health (pp. 5–24). San fran-cisco, CA: Wiley.

Nastasi, B. K., & hitchcock, J. (2009). Challenges of evaluating multilevel interventions. American Journal of Community Psychology, 34(4), 360–376.

Nelson, G., Janzen, R., Ochocka, J., & trainor, J. (2010). participatory action research and evaluation with mental health self-help groups and or-ganizations: A theoretical framework. in l. Brown & S. Wituk (Eds.),


Mental health self-help: Consumer and family driven initiatives (pp. 39–58). New york, Ny: Springer.

Nelson, G., Ochocka, J., Griffen, K., & lord, J. (1998). “Nothing about me, without me”: participatory action research with self-help/mutual aid organizations for psychiatric consumer/survivors. American Journal of Community Psychology, 26, 881–912.

Nelson, G., Ochocka, J., Janzen, R., trainor, J., & lauzon, S. (2004). A comprehensive evaluation framework for mental health consumer/survivor organizations: values, conceptualization, design, and action. Canadian Journal of Program Evaluation, 19(3), 29–53.

Newbrough, J. R. (1995). toward community: A third position. American Journal of Community Psychology, 23(1), 9–37.

Ochocka, J., moorlag, E., & Janzen, R. (2010). A framework for entry: pAR values and engagement strategies in community research. Gateways: International Journal of Community Research & Engagement, 3, 1–19.

patton, m. Q. (2008). Utilization-focused evaluation (4th ed.). Beverly hills, CA: Sage.

patton, m. Q. (2011). Developmental evaluation: Applying complexity con-cepts to enhance innovation and use. New york, Ny: Guilford press.

public policy forum. (2004). Bring employers into the immigration debate. Ottawa, ON: Author.

Reason, p. (2006). Choice and quality in action research practice. Journal of Management Inquiry, 151(2), 187–203.

Shadish, W. R., Cook, t. d., & leviton, l. C. (1991). Foundations of program evaluation: Theories of practice. Newbury park, CA: Sage.

Stake, R. (1978). the case study method of social inquiry. Educational Re-searcher, 7(2), 5–8.

Stringer, E. t. (2007). Action research (3rd ed.). thousand Oaks, CA: Sage.

tashakkori, A., &. teddlie, C. (2003). the past and future of mixed methods research: from data triangulation to mixed model designs. in A. tashakkori & C. teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 671–701). thousand Oaks, CA: Sage.


taylor, A., & Botschner, J. (1998). Evaluation handbook. Kitchener, ON: Centre for Community Based Research (formerly Centre for Re-search and Education in human Services).

travers, R., Wilson, m. G., flicker, S., Guta, A., Bereket, t., mcKay, C., … Rourke, S. B. (2008). the greater involvement of people living with AidS principle: theory versus practice in Ontario’s hiv/AidS com-munity-based research sector. AIDS Care, 20(6), 615–624.

Wallerstein, N., & duran, B. (2003). the conceptual, historical, and practice roots of community based participatory research and related partici-patory traditions. in m. minkler & N. Wallerstein (Eds.), Community-based participatory research for health (pp. 25–46). San francisco, CA: Wiley.

Wayland, S. (2006). Unsettled: Legal and policy barriers for newcomers to Canada. Ottawa, ON: Community foundations of Canada and the law Commission of Canada.

Wayland, S. (2007). The Learning Exchange: Disseminating good ideas: Skilled immigrants in the labour market. Report prepared for CiC and the J.W. mcConnell foundation.

Rich Janzen is Research director at the Centre for Community Based Research (www.communitybasedresearch.ca). Rich recently completed his ph.d. (Community psychology) at Wilfrid laurier University.

DanielaSeskar-Hencicis the former manager of planning, Evalu-ation and Epidemiology at the Regional municipality of Waterloo. presently daniela is Associate director, institutional Analysis and planning at the University of Waterloo.

Yasir Dildar is Senior Researcher at the Centre for Community Based Research. yasir has m.A.s in sociology and development stud-ies.

PeterMcFaddenis the former Executive director of the Waterloo Region immigrant Employment Network (WRiEN).

using evaluation to shape and direct comprehensive ... · yasir dildar centre for community based...

Documents