how do you evaluate a network? a canadian child and …

123

Corresponding author: J.K. Popp, Director, Southern Alberta Child and Youth Health Network, Room 3517, c/o Alberta Children’s Hospital, 1820 Richmond Rd. SW, Calgary, AB T2T 5C7; <[email protected]>

HOW DO YOU EVALUATE A NETWORK? A CANADIAN CHILD AND YOUTH HEALTH NETWORK EXPERIENCE

Janice K. PoppLaura N. L’HeureuxSouthern Alberta Child and Youth Health NetworkCalgary, Alberta

Carly M. DolinskiCalgary Health RegionCalgary, Alberta

Carol E. AdairUniversity of CalgaryCalgary, Alberta

Suzanne C. ToughCalgary Health RegionCalgary, Alberta

Ann L. CasebeerUniversity of CalgaryCalgary, Alberta

Kathleen L. Douglas-EnglandCalgary Health RegionCalgary, Alberta

Catherine C. MorrisonSouthern Alberta Child and Youth Health NetworkCalgary, Alberta

Over the past decade, approximately 20 child and youth health networks have been initiated in Canada. The value of any net-work depends on its effectiveness in achieving stated goals; how-ever, description and measurement of network effectiveness is challenging, given the complex multi-sectoral and/or multidisci-plinary relationships involved and the relative dearth of evalua-tion methods specifi c to networks. Since its inception in 2001, the Southern Alberta Child and Youth Health Network (SACYHN) has reviewed the network evaluation literature and developed

The Canadian Journal of Program Evaluation Vol. 20 No. 3 Pages 123–150ISSN 0834-1516 Copyright © 2005 Canadian Evaluation Society

Abstract:

THE CANADIAN JOURNAL OF PROGRAM EVALUATION124

and implemented an evaluation framework. This article describes key fi ndings from the literature on networks and their evaluation, and the experience and learning from network evaluation activi-ties conducted by SACYHN. These may inform evaluation efforts in other networks or similar inter-organizational initiatives.

Au cours de la dernière décennie, le Canada a connu l’établisse-ment d’environ 20 réseaux pour la santé des enfants et des jeunes. La valeur de tout réseau dépend de son effi cacité à atteindre les buts précisés; toutefois, la description et la mesure de l’effi cacité du réseau posent tout un défi étant donné les relations plurisec-torielles complexes et/ou pluridisciplinaires dont il est question et la pénurie relative de méthodes d’évaluation précises pour les réseaux. Depuis sa création en 2001, le Southern Alberta Child and Youth Health Network (SACYHN) a étudié la littérature sur l’évaluation des réseaux et a préparé un cadre d’évaluation et l’a mis en oeuvre. Cet article décrit les conclusions principales tirées des études sur les réseaux et leur évaluation ainsi que l’expé-rience et les leçons tirées des activités d’évaluation des réseaux réalisées par le SACYHN. Ces constatations pourront informer les efforts d’évaluation dans d’autres réseaux ou des initiatives inter-organisationnelles semblables.

Canadian communities face challenges in delivering health and related services to children and youth as a consequence of limited availability of specialty paediatric expertise, competition for scarce resources within largely adult-focused service systems, a need to provide access to specialized services over large geographic areas, multiple organizational mandates of different child-serving organizations and sectors, and the necessity of considering a child’s health and well-being within the family and environmental context. To address these challenges, organizations are using networks to develop shared visions, collaborative relationships, and joint initia-tives (Weiss, Anderson, & Lasker, 2002). A network is “a grouping of individuals, organizations, and agencies on a non-hierarchical basis around common issues or concerns, which are pursued proactively and systematically based on commitment and trust” (World Health Organization, 1998) and which include attachment to a common purpose (Chisholm, 1998).

In Canada, at least 20 known child and youth health networks have been established to attend to some or all of the following functions: provision/coordination/planning of services, policy development, pri-ority setting, research, communication and information sharing. These networks may focus exclusively on health care or include other

Résumé:

125LA REVUE CANADIENNE D’ÉVALUATION DE PROGRAMME

child-serving sectors such as education and social services, and vary in scope from local to regional to provincial (Southern Alberta Child and Youth Health Network, 2004). By promising to address complex child health issues through a voluntary multisectoral and multidisci-plinary approach with a horizontal rather than hierarchical structure, networks face daunting challenges in relation to evaluating their functioning and in demonstrating that they add value beyond basic organizational connection or communication. One such network cur-rently experiencing these challenges is described below.

Established in September 2001, the Southern Alberta Child and Youth Health Network (SACYHN) is a dynamic, voluntary, cross-sector collaboration that geographically spans four health regions across southern Alberta and is concerned with the health and well-being of children, youth, and families (see Figure 1). Its focus is on creating inter-sectoral and inter-regional linkages that will increase capac-ity to respond seamlessly to the needs of children and youth (Popp, Douglas-England, Casebeer, & Tough, 2005). SACYHN membership includes families (both parents and youth), government ministries, provincial organizations, regional authorities, First Nations, and not-for-profi t agencies, all from a variety of child-serving sectors such as health, education, justice, and children’s services. Representa-tives from these groups participate in various committees, working groups, and regional groups. While families participate in many ways, parents have specially designated representation on the SACYHN Steering Committee and youth participate primarily through the Child and Youth Advisory Council. A small number of SACYHN staff and regional network coordinators manage the key areas of SACYHN activity, such as outreach services and telehealth, a family and com-munity resource centre, primary care capacity-building projects, communications, and evaluation. Funding for SACYHN activities and initiatives comes from direct contributions of partners to core network functions, in-kind support, joint cost-sharing on specifi c initiatives, and proposals to external agencies.

Given the complex structure and context described above, how can SACYHN demonstrate that it is an effective mechanism for address-ing equally complex issues in delivering health and related services to children and youth? The purpose of this article is to describe key fi ndings from the literature on networks and their evaluation, and the experience and learning from network evaluation activities conducted by SACYHN. These may inform evaluation efforts in other networks or similar inter-organizational initiatives.


Stee

rin

gC

om

mit

tee

Reg

ion

alIn

ter-

sect

ora

lC

om

mit

tees

Wo

rkin

gG

rou

ps

SAC

YHN

Dir

ecto

rate

Net

wor

k C

oo

rdin

ato

rs

Ou

r Vis

ion

The

hea

lth

an

d w

ell-

bei

ng

of c

hild

ren

,yo

uth

an

d fa

mili

es is

su

pp

ort

ed b

y h

igh

qu

alit

y, c

oo

rdin

ated

pro

gra

ms,

serv

ices

and

info

rmat

ion

reso

urc

es t

hat

are

acce

ssib

le a

nd

as

clo

se to

ho

me

asp

oss

ible

.

Prov

ides

str

ateg

ic d

irec

tion

and

advo

cacy

for S

AC

YHN

Inte

r-se

ctor

al g

roup

s th

at id

enti

fy a

ndw

ork

on p

rior

itie

s w

ithi

n th

eir

geog

raph

ic re

gion

s in

kee

ping

wit

hSA

CYH

N v

isio

n an

d m

issi

on

Task

ori

ente

d gr

oups

that

und

erta

kesp

ecifi

c SA

CYH

N p

roje

cts/

stra

tegi

es(c

ross

-reg

iona

l and

/or c

ross

-sec

tora

l)

Figu

re 1

SACY

HN S

chem

atic


FINDINGS FROM THE LITERATURE ON NETWORKS AND THEIR EVALUATION

A literature review on networks undertaken by SACYHN (Hill, 2002) uncovered general characteristics of networks, specifi c characteristics that challenge network evaluation, and proposed evaluation frame-works. Examples of implementing evaluation frameworks were rare. An important conclusion of the literature review was that “network studies tend to focus on network composition and function rather than elements crucial to meeting network goals or developing the vision” (Hill, p. 43).

General Characteristics of Networks

Networks are identifi ed by their voluntary connections and relation-ships among systems, organizations, and individuals. Individuals or organizations enter into network relationships at their own desired level and intensity, allowing them to maintain their autonomy. A network can have members who are strongly tied together as well as others who are more loosely connected but still bound through a commitment to a core vision (Chisholm, 1996). All members contribute to the overall strength and effectiveness of the network. In general, the greater the number and variety of connections, the more likely the network will have the capacity to creatively tackle complex is-sues, explore innovations, and sustain itself over time (Provan & Milward, 2001).

Provan and Milward (2001) describe effective networks as having the following characteristics:

• multiple levels of involvement (within organizations, front-line staff to senior management; local to regional to provin-cial);

• focused integration (targeted and appropriate linkages among individuals, organizations, communities, and systems);

• tie strength (interdependence among key members across multiple types of involvement);

• network governance (in a form necessary to sustain a net-work);

• involvement (interaction based on trust and commitment to network goals);

• legitimacy (both within the network and in the larger com-munity or system);


• resources (suffi cient to build and sustain the network);• goals (specifi c, attainable, and appealing to members); and • stability (over time with no major or frequent system upheav-

als).

Chisholm (1996) identifi es four functions of effective networks: (a) creating and maintaining a vision that binds partner organizations together, (b) serving as a forum for dealing with complex issues, (c) identifying the importance of attitudes and perceptions for broad development, and (d) providing ways of communicating. Critical to these network functions are the underlying elements of mutual trust, willingness and desire to facilitate change, interest in gaining legitimacy among network agencies and within the community, and political power.

Networks are also described as learning organizations that are self-aware and self-regulating. To generate this self-knowledge, communi-cation and continuous evaluation are critical components of network functioning (Mays, Halverson, & Kaluzny, 1998).

Characteristics of Networks That Make Them Challenging to Evaluate

Studies evaluating the effectiveness of networks are few. This may be attributed to the nebulous nature of networks, perceptions of connectedness, role confusion, divergence in defi ning criteria for success, and the diffi culty in identifying measurable outcomes, as detailed below.

The nebulous nature of networks poses challenges for evaluation (Rose, 2004). As described above and in more detail in Hill (2002), networks are a complex arrangement of multisectoral/multi-agency and/or multidisciplinary relationships with shared processes and objectives. Provan and Milward (2001) articulate this as the “joint production problem,” where multiple agencies are responsible for one or more components of a single service creating blurred or indistinct organizational boundaries. This unifi ed delivery of services may sat-isfy clients, but presents challenges to the attribution of outcomes.

According to Provan, Veazie, Staten, and Teufel-Shone (2005), vari-ation in connectedness of network relationships is desirable. A mix of strong and weak ties improves effi ciency and allows for additional sources of information and ideas to be forthcoming. The value of strong ties is that members are signifi cantly and consistently invested


in multiple network initiatives, whereas the value of weak ties is that members bring a potentially broader or unique perspective. Though this mix is attractive, some network participants, particularly those who are weakly tied, may not self-identify as network members, even when recognized by the network as such. This incongruence in perceived member status is a challenge for network evaluation, as data collection and measurement can be impacted if those with weak ties do not see their relevance to the evaluation and decline to participate (Dolinski, 2005; Provan et al., 2005). The result is a lack of understanding of the perceptions and experiences of more weakly tied members, which in turn impedes achieving a complete under-standing of how the overall network is or is not effective.

Other impediments to the evaluation of networks are related to con-fusion about roles, and diverging views of success and health. First, differences in opinion amongst network leaders and members as to how the overall work done by networks should be defi ned, articu-lated, and implemented (Child and Youth Health Networks, 2003) can hamper assessment of effectiveness due to lack of agreement on what should be evaluated and how success should be defi ned (Provan et al., 2005). Second, networks and other collaborative partnerships often seek outcomes in multiple domains, and improvement in any given domain may be defi ned differently by individual participants, thus challenging the development of evaluation criteria (Provan et al., 2005; Taylor-Powell, Rossing, & Geran, 1998). Finally, traditional health outcome indicators are often medically defi ned and therefore clinical in nature. Many child and youth health networks subscribe to a broader defi nition of health (SACYHN, 2004). Consequently, appro-priate measurable outcome indicators that encompass this vision need to be identifi ed, potentially increasing the complexity of evaluation.

All of these factors that contribute to network complexity suggest the need for a correspondingly complex or systemic approach to network evaluation.

Proposed Evaluation Frameworks: Developing a Network Evaluation Matrix

The challenges to evaluating inter-organizational partnerships and collaborations are not new. However, our literature review uncovered only one framework (that of Provan & Milward, 2001) that attempted to evaluate a complex arrangement of partnerships and collaborations — that is, networks — with a comparably sophisticated multidimen-


sional approach. This multidimensional framework fi gures heavily into a broader systems approach to evaluation that identifi es levels of analysis and associated evaluation outcomes as culled from the literature. The resulting network evaluation matrix can be used to assess the effectiveness of network processes and activities from multiple stakeholder perspectives.

Networks operate on four levels (vision, structure, process, and service delivery) and thus network evaluation must pay attention to func-tioning at each level:

1. The overarching operational concern is vision (Mays et al., 1998), as it is the vision that binds the participants together (Chisholm, 1998). Therefore, networks need to develop a clear and well-articulated vision early in their formation.

2. The second level involves structure, or how the network is arranged (Bolland & Wilson, 1994; Gray & Wood, 1991; Mays et al., 1998). Structural functions include assembling, ad-ministering, evolving, regulating, maintaining, and funding the network, as well as planning activities. Issues associated with the structure include the need for an identifi ed form, network composition, consideration of stakeholders and end-users, resources, technical assistance, and e-health.

3. A third level is that of process, or how the network fulfi lls its vision (Gray & Wood, 1991; Mays et al., 1998). Proc-ess functions include confi guring, interacting, coordinating, learning, sensing, developing, deliberating, confl icting, com-municating, planning, collaborating, surveying, and evaluat-ing. Issues associated with process include trust, the need to develop and sustain relationships, time, managing confl ict, communication strategies, and the need for quick wins.

4. The fourth level of network operation is service delivery, which includes exchanging information, raising public aware-ness, case management, co-locating, establishing a program resource centre, and standardizing referral and intake proc-esses (Bolland & Wilson, 1994). Issues associated with service delivery include the need to plan regarding systems integra-tion, the need for joint funding, and the immense challenges of integrated service delivery.

In addition, networks have impacts at different levels and therefore must also be evaluated on each of these (community, network, and organization/participant) to capture adequately the perspectives of their multiple and diverse stakeholders (Provan & Milward, 2001):


1. The community level involves the impact of network activi-ties and processes on stakeholder groups such as client ad-vocacy groups, funders, regulators, and the general public. Outcomes associated with this level of analysis may include cost to community, building social capital, public perceptions that the problem is being solved, changes in incidence of the problem, and aggregate indicators of client well-being (Provan & Milward, 2001).

2. The network level is about whether stakeholder groups rep-resented by primary funders and regulators, network admin-istrative organizations (NAO), and member organizations believe that network processes and initiatives are successful. Outcomes may include network membership growth, range of services provided, absence of service duplication, relation-ship strength (or multiplexity), creation and maintenance of NAO, integration and coordination of services, cost of network maintenance, and member commitment to network goals (Provan & Milward, 2001).

3. Provan and Milward’s (2001) third level to consider, the organization/participant level, can be separated into the distinct levels of individual and organizational, an approach favoured by SACYHN. Both levels include the perspective of stakeholders such as the member agency board and manage-ment, agency staff, or individual clients. Outcomes associated with the individual level include service access, client level health outcomes, and staff outcomes, whereas organizational outcomes include agency survival, enhanced legitimacy, re-source acquisition, service costs, member satisfaction, refer-rals, and collaborative attitudes.

Therefore, a comprehensive assessment of network effectiveness should consider the level on which the network is operating and the varying levels on which the impact is felt, both of which provide the basis for our own evaluation matrix (see Table 1). Networks that seek, adhere to, and measure a well-articulated vision and forge trustful and committed relationships among network participants are more likely to operate at numerous levels and impact on multiple and diverse stakeholders (Hill, 2002).

While the literature on network extinction or failure is scant (Hill, 2002), what is there suggests that issues related to network disband-ment should be included in their monitoring and measurement. Ab-sence of a clear vision (Fawcett, Francisco, Paine-Andrews, & Schultz,

THE CANADIAN JOURNAL OF PROGRAM EVALUATION132Ta

ble

1Ne

twor

k Ev

alua

tion

Mat

rix w

ith S

ampl

e Ac

tiviti

es a

nd In

dica

tors

for S

ACYH

N

Le

vel o

f im

pact

and

sam

ple i

ndica

tors

Leve

l of n

etwor

k ope

ratio

n Co

mm

unity

Ne

twor

k Or

gani

zatio

nal

Indi

vidua

l

Visio

n

Stru

cture

Proc

ess

Serv

iceDe

liver

y

Sour

ce: A

dapt

ed fr

om H

ill (2

002)

.* I

ndica

tors

relat

ed to

netw

ork d

isban

dmen

t/exti

nctio

n.

• Sh

ared

und

ersta

ndin

g of

pu

rpos

e and

goa

ls

• Ar

rang

emen

t or c

ompo

sitio

n of

the n

etwor

k•

Activ

ities

inclu

de as

sem

-bl

ing,

fund

ing,

adm

inist

er-

ing,

regu

latin

g, et

c.

• W

hat t

he n

etwor

k doe

s to

fulfi

ll its

visio

n•

Activ

ities

inclu

de co

ordi

nat-

ing,

lear

ning

, dev

elopi

ng,

com

mun

icatin

g, et

c.

• Sh

ared

deli

verin

g of

serv

ices

and/

or fa

cilita

tion

of se

rvice

de

liver

y •

Activ

ities

inclu

de ca

se

man

agem

ent,

exch

angi

ng

info

rmati

on, r

eferra

l and

in

take,

etc.

• Po

licies

alig

n vis

ion

and

miss

ion

• Po

licy d

evelo

pmen

t an

d ad

optio

n

• Po

licy i

mpl

emen

tatio

n•

Com

mun

icatio

n m

ateria

ls

• Ex

posu

re•

Abse

nce/

redu

ction

of

serv

ice d

uplic

ation

• Af

fi liat

ion

Agre

emen

t sig

ned

• Pe

rcep

tions

that

netw

ork l

acks

us

efuln

ess*

• Co

hesio

n•

Cent

ralit

y•

Relat

ions

hip

stren

gth

(or m

ulti-

plex

ity)

• De

velo

pmen

t and

gro

wth

• Co

llabo

rativ

e atti

tude

s•

Cost

over

runs

*•

Stak

ehol

der t

urno

ver*

• St

ructu

re•

Delib

erati

on•

Lack

of t

rust*

• La

ck o

f com

mitm

ent*

• In

suffi

cient

tim

e to

build

relat

ion-

ship

s*

• Sh

arin

g in

form

ation

(e.g

., co

nfer

-en

ces,

news

letter

s, m

eetin

gs)

• In

abili

ty to

chan

ge se

rvice

*•

Inab

ility

to re

alloc

ate m

ainstr

eam

re

sour

ces*

• Se

rvice

s alig

n vis

ion

and

miss

ion

• Su

cces

sful j

oint

pro

gram

s, pr

ojec

ts•

Agre

emen

ts, p

roto

cols,

pr

ogra

m p

olici

es•

Shar

ed cl

ient b

ase

• Sh

ared

data

base

• Pa

rticip

ation

in n

etwor

k pr

ojec

ts

• Ap

prop

riate

refer

rals

• In

crea

sed

awar

enes

s•

Incr

ease

d kn

owled

ge

• Pa

rent

inpu

t (at

Stee

ring

Com

mitt

ee le

vel)

• Pa

rent

and

yout

h in

put (

at W

orkin

g Gr

oup

level)

• Co

llabo

rativ

e atti

tude

s

• En

hanc

ed kn

owled

ge o

f ch

ild h

ealth

• Sh

arin

g in

form

ation

with

pa

rent

s/ fa

mili

es•

Impr

oved

conn

ectio

ns


2000); changes in leadership or stakeholder representation (Fawcett et al., 2000; Fonner, 1998); perceptions that networks lack usefulness (Chisholm, 1998); early confl ict, lack of time to build relationships, and inability to effect changes in services (Goss, 2001); and discomfort with an ambiguous structure (Gilchrist, 1995) are some of the issues associated with network failure. Investigating these dimensions in network evaluation should help networks avoid major pitfalls in their ongoing development.

Overall, few studies reported in the literature evaluate networks using multiple levels of analysis (Hill, 2002), and the lack of data linking networks to outcomes (Lehman, Postrado, Roth, McNary, & Goldman, 1994; Provan & Milward, 2001) limits understanding of how networks may be mechanisms for addressing complex health issues (O’Toole, 1997).

EVALUATION EXPERIENCE OF SACYHN

SACYHN has undertaken or been involved in a number of evalua-tion activities that demonstrate how it has evolved its approach to assessing network effectiveness.

In its fi rst year, in addition to undertaking a network literature re-view, SACYHN participated in an external research project on change in health care delivery which involved more than 30 interviews with SACYHN stakeholders between September 2001 and December 2002 (Rose, 2004). This formative evaluation provided SACYHN with feed-back on its position in the community, how the vision and mission were perceived, and what issues would be important for sustainability. The results challenged SACYHN to consider how best to achieve its initiatives. That is, should networks prioritize building network infra-structure in an attempt to accomplish goals, or invest in the creation of network processes (e.g., building community capacity) so that the work can be done differently (and more effectively)? Rose (2004) also highlighted the need for network strategies on how to facilitate the diffi cult conversations that will arise about membership agreement and changes, outcome discrepancies, process discrepancies, and un-intended consequences of action, both positive and negative.

In 2003, to encourage a national dialogue about network models, evaluation strategies, and outcomes, SACYHN prepared and dis-tributed a National Network Survey through the Canadian Associa-tion of Paediatric Health Centres general membership mailing list


(SACYHN, 2004). Thirteen questionnaires were returned, represent-ing 12 unique child and youth health networks. Respondents most frequently reported that evaluation activities were conducted in relation to specifi c projects (53.8%) or focused on member satisfaction (46.2%). The survey also revealed that only three networks (23.1%) believed they had evaluation instruments that would be useful for other networks.

Development of the SACYHN Implementation Evaluation Framework

Coinciding with the formative evaluation and subsequent to the SACYHN literature review, SACYHN developed an Implementa-tion Evaluation Framework (Figure 2) to guide evaluation efforts. The framework identifi ed activity/process (i.e., output) and process and health outcome (i.e., success) indicators from the evaluation matrix derived from the literature. The indicators were articulated in reference to level of impact (individual, organizational, network, or community), thereby promoting multiple levels of analysis within evaluation of the network.

The Implementation Evaluation Framework was titled as such to ac-knowledge that evaluation activities of the network might vary de-pending on the developmental phase. For example, evaluation early in a network’s development might focus on engagement or the degree to which the vision is shared, a critical step in network development; whereas in a mature network, which the literature suggests can take several years to achieve (Dockery, 1996; Goss, 2001), the concrete bene-fi ts of the network should be clearer and one might expect evaluation to identify actual outcomes or the degree to which the network has met its goals (Hill, 2002). Thus, evaluation activities appropriate to a network in its early development may not fi t for a mature network. The point that evaluation must account for evolution has also been made in literature on collaborative partnerships (Taylor-Powell et al., 1998).

Of note, the Implementation Evaluation Framework also considered the relational and technical quality of the SACYHN vision (an opera-tional level of analysis), and acknowledged the linkages and collabora-tions required to achieve that vision. SACYHN’s vision is: “The health and well being of children, youth and families is supported by high quality, coordinated programs, services and information resources that are accessible and as close to home as possible.” The framework defi ned the following four critical elements to capture the relational quality of SACYHN’s vision:


Figu

re 2

SACY

HN Im

plem

enta

tion

Eval

uatio

n Fr

amew

ork

(pre

sent

ed in

par

t: Ne

twor

k go

als

1 &

2 o

nly)

Mis

sio

n:

The

Net

wo

rk w

ill u

se it

s co

llect

ive

stre

ng

th &

exp

erti

se t

o b

uild

& s

ust

ain

lin

kag

es t

hat

op

tim

ize

the

hea

lth

& w

ell b

ein

g o

f all

child

ren

, yo

uth

& fa

mili

es.

SAC

YH

N V

isio

nTh

e h

ealt

h a

nd

wel

l-b

ein

go

f ch

ildre

n, y

ou

th &

fam

ilies

is s

up

po

rted

by:

Co

mp

on

ents

/Dim

ensi

on

sN

etw

ork

Go

als

Year

s 1

to

3

Hig

h qu

alit

y p

rog

ram

s,se

rvic

es &

info

rmat

ion

re

sou

rces

Coo

rdin

ated

pro

gra

ms,

serv

ices

& in

form

atio

n

reso

urc

es

Acc

essi

ble

pro

gra

ms,

serv

ices

& in

form

atio

nre

sou

rces

Rela

tio

nal

Qu

alit

y:

- Ch

ild-f

ocu

sed

& fa

mily

-cen

tred

(fam

ilies

as

par

tner

s, sh

arin

g in

form

atio

n,

stre

ng

ths

bas

ed, r

esp

ectf

ul,

ho

no

ur d

iver

sity

, rec

og

niz

e d

evel

op

men

tal

nee

ds,

flexi

ble

ser

vice

s, cu

ltu

rally

sen

siti

ve, r

esp

on

sive

)

- Rec

og

nit

ion

of u

niq

ue

hea

lth

nee

ds

of A

bo

rig

inal

fam

ilies

- S

up

po

rt &

val

ue

all s

ervi

ce p

rovi

der

s

- Fo

cus

on

bro

ader

det

erm

inan

ts o

f hea

lth

Tech

nic

al Q

ual

ity:

- S

kille

d s

ervi

ce p

rovi

der

s

- Co

nsi

sten

t st

and

ard

s o

f car

e

- Evi

den

ce b

ased

car

e

- Co

nti

nu

ou

s im

pro

vem

ent/

on

go

ing

eva

luat

ion

Lin

kag

es/C

olla

bo

rati

on

(pla

nn

ing,

pri

ori

ty s

etti

ng,

ser

vice

del

iver

y):

- B

etw

een

pro

gra

ms/

serv

ices

- B

etw

een

ag

enci

es

- Bet

wee

n s

ecto

rs

- Bet

wee

n re

gio

ns

- B

etw

een

pro

vid

ers

- B

etw

een

fam

ilies

- B

etw

een

fam

ilies

& p

rovi

der

s

- In

-per

son

&/o

r ele

ctro

nic

- M

ult

iple

mo

des

of a

cces

s (e

.g.,

lan

gu

ages

, tel

ehea

lth

)

- Mu

ltip

le p

oin

ts o

f acc

ess

- M

ult

iple

refe

rral

pat

hs

(e.g

., se

lf, p

rovi

der

)

- Clo

ser t

o h

om

e

- Aff

ord

able

- A

deq

uat

e co

mm

un

ity

cap

acit

y

1. D

evel

op

& a

do

pt

a sh

ared

vis

ion

& m

issi

on

2. Id

enti

fy &

cre

ate

op

po

rtu

nit

ies

for p

osi

tive

c

han

ge

in s

ervi

ce d

eliv

ery

3. In

volv

e fa

mili

es &

ser

vice

pro

vid

ers

in

s

ign

ifica

nt

way

s in

ser

vice

pla

nn

ing

&

del

iver

y

4. B

uild

& s

ust

ain

inte

r-re

gio

nal

, cro

ss-

s

ecto

ral &

clin

ical

co

nn

ecti

on

s

5. D

evel

op

or i

mp

rove

ele

ctro

nic

lin

kag

es

THE CANADIAN JOURNAL OF PROGRAM EVALUATION136Fi

gure

2SA

CYHN

Impl

emen

tatio

n Ev

alua

tion

Fram

ewor

k (p

rese

nted

in p

art:

Netw

ork

goal

s 1

& 2

onl

y) —

con

tinue

d

Net

wo

rk

Go

al #

1M

eth

od

So

urc

e o

f In

form

atio

n

Act

ivit

y/P

roce

ss(O

utp

ut)

Ind

icat

ors

Pro

cess

& H

ealt

h O

utc

om

e

In

dic

ato

rs (S

ucc

ess)

Dev

elo

p &

ado

pt

ash

ared

vis

ion

& p

urp

ose

- Ag

reed

up

on

su

stai

nab

ility

pla

n

(lon

g te

rm fi

nan

cial

su

pp

ort

&

ov

ersi

gh

t)- S

har

ed re

sou

rcin

g /

will

ing

nes

s to

re

sou

rce

wo

rk o

f th

e N

etw

ork

- Su

stai

ned

mem

ber

ship

(lo

w t

urn

over

rate

)- I

ncr

easi

ng

# o

f mem

ber

s ov

er t

ime

- In

crea

sin

g #

of s

ecto

rs in

volv

ed- I

ncr

ease

d in

ter-

reg

ion

al c

olla

bo

rati

on

- In

crea

sed

cro

ss-s

ecto

ral c

olla

bo

rati

on

- Par

tici

pan

t sa

tisf

acti

on

(85%

tar

get

)

(Net

wor

k &

org

aniz

atio

nal l

evel

out

com

es)

- In

crea

sed

co

mm

un

ity

& a

gen

cy

u

nd

erst

and

ing

of t

he

Net

wor

k’s

visi

on

& m

issi

on

- Co

nsi

sten

cy o

f mes

sag

es a

bo

ut

the

N

etw

ork

- In

crea

sed

cro

ss-s

ecto

ral r

esp

on

sib

ility

for c

hild

& y

ou

th h

ealt

h- I

ncr

ease

d p

arti

cip

atio

n in

Net

wor

k

wo

rkin

g g

rou

ps,

com

mit

tees

& p

roje

cts

- In

crea

sed

lin

kag

es w

ith

pro

vin

cial

& n

atio

nal

init

iati

ves

- In

crea

sed

lin

kag

es w

ith

oth

er c

hild

&

yo

uth

hea

lth

net

wo

rks

(Net

wor

k &

com

mun

ity

leve

l out

com

es)

- # o

f Sig

nat

ori

es to

Aff

iliat

ion

Ag

reem

ent

- Net

wor

k ac

tio

n p

lan

dev

elo

ped

- # o

f reg

ion

al N

etw

ork

com

mit

tees

form

ed- R

egio

nal

act

ion

pla

ns

dev

elo

ped

- # o

f Net

wor

k C

oo

rdin

ato

rs a

pp

oin

ted

(by

reg

ion

& s

ecto

r)- N

etw

ork

invo

lvem

ent

in re

sear

ch o

n

o

rgan

izat

ion

al c

han

ge

in t

he

hea

lth

care

sec

tor (

Dr.

Tere

sa R

ose

)

- Co

mm

un

icat

ion

pla

n d

evel

op

men

t &

im

ple

men

tati

on

- # o

f pre

sen

tati

on

s o

r mee

tin

gs

wit

h

ke

y st

akeh

old

ers

& b

road

co

mm

un

ity

- Bro

chu

re /

new

slet

ter d

evel

op

men

t- P

ost

er p

rese

nta

tio

n m

ater

ials

- Net

wor

k w

ebsi

te- #

of a

rtic

les

or n

ewsl

ette

r ite

ms

p

ub

lish

ed a

bo

ut

the

Net

wor

k- C

om

ple

tio

n o

f lit

erat

ure

revi

ew o

n

n

etw

ork

s- P

ub

licat

ion

of a

rtic

les

fro

m li

tera

ture

revi

ew- P

rovi

nci

al &

nat

ion

al li

nka

ges

esta

blis

hed

- Ste

erin

g C

om

mit

tee

- Reg

ion

al N

etw

ork

C

om

mit

tees

- Net

wor

k C

oo

rdin

ato

rs- W

ork

ing

Gro

up

s

- Co

mm

itte

e

par

tici

pat

ion

- Co

mm

un

icat

ion

- Co

mm

un

icat

ion

W

ork

ing

Gro

up

- Eva

luat

ion

Wo

rkin

g

Gro

up

- Ste

erin

g C

om

mit

tee


Figu

re 2

SACY

HN Im

plem

enta

tion

Eval

uatio

n Fr

amew

ork

(pre

sent

ed in

par

t: Ne

twor

k go

als

1 &

2 o

nly)

— c

ontin

ued

Net

wo

rk

Go

al #

2M

eth

od

So

urc

e o

f In

form

atio

n

Act

ivit

y/P

roce

ss(O

utp

ut)

Ind

icat

ors

Pro

cess

& H

ealt

h O

utc

om

e

In

dic

ato

rs (S

ucc

ess)

Iden

tify

& c

reat

eo

pp

ort

un

itie

s fo

rp

osi

tive

ch

ang

ein

ser

vice

del

iver

y

- Im

pro

ved

acc

essi

bili

ty o

f car

e (c

lose

r

to h

om

e; le

ss t

rave

l, co

st &

fam

ily

st

ress

)- 1

00%

of r

egio

ns

& s

ervi

ces

usi

ng

Ou

trea

ch S

ervi

ces

Fram

ewo

rk- I

ncr

ease

d c

linic

al c

on

nec

tio

ns

acro

ss

re

gio

ns

- In

crea

sed

co

nti

nu

ity

of c

are

- In

crea

sin

g #

of f

orm

al N

etw

ork

m

emb

ers

over

tim

e- I

ncr

ease

d in

ter-

reg

ion

al c

olla

bo

rati

on

- In

crea

sed

cro

ss-s

ecto

ral c

olla

bo

rati

on

- Im

pro

ved

un

der

stan

din

g o

f bu

sin

ess

p

roce

sses

& a

ctiv

itie

s ac

ross

sec

tors

- In

crea

sed

alig

nm

ent

of p

olic

ies

ac

ross

reg

ion

s &

sec

tors

- Do

cum

ente

d p

ract

ice

chan

ges

(Net

wor

k &

org

aniz

atio

nal l

evel

out

com

es)

- Su

cces

sfu

l res

ou

rcin

g o

f pro

ject

s- I

mp

lem

enta

tio

n o

f pro

ject

s- N

ew p

artn

ersh

ips

dev

elo

ped

- En

chan

ced

pro

fess

ion

al c

apac

ity

(Net

wor

k &

org

aniz

atio

nal l

evel

out

com

es)

- Dev

elo

pm

ent

& a

do

pti

on

of O

utr

each

Se

rvic

es F

ram

ewo

rk- #

of c

linic

al a

reas

usi

ng

Ou

trea

ch

Se

rvic

es F

ram

ewo

rk- #

of r

egio

ns

usi

ng

Ou

trea

ch S

ervi

ces

Fr

amew

ork

- Tel

ehea

lth

lau

nch

- # o

f org

aniz

atio

ns

sig

nin

g t

he

A

ffili

atio

n A

gre

emen

t- P

lan

nin

g &

pri

ori

ty s

etti

ng

acr

oss

sect

ors

wit

hin

reg

ion

s

- Id

enti

ficat

ion

of r

egio

nal

& s

ecto

r

pri

ori

ties

for a

ctio

n- I

den

tific

atio

n o

f nee

ds,

gap

s &

edu

cati

on

al p

rio

riti

es- N

etw

ork

pro

ject

dev

elo

pm

ent

(s

ervi

ce/p

rog

ram

, lo

cal/

reg

ion

al,

b

road

Net

wor

k le

vel,

cro

ss-s

ecto

ral)

- Ou

trea

ch S

ervi

ces

Fr

amew

ork

- Tel

ehea

lth

init

iati

ve- A

ffili

atio

n A

gre

emen

t- R

egio

nal

Net

wor

k

Co

mm

itte

es

- Po

licy

d

evel

op

men

t

- Co

mm

itte

e

par

tici

pat

ion

- Ste

erin

g C

om

mit

tee

- Reg

ion

al N

etw

ork

C

om

mit

tees

- Sta

keh

old

er m

eeti

ng

s

(CEO

s, B

oar

ds,

co

mm

un

ity,

etc

).

*Net

wor

k go

als

#3,

4 &

5 a

ll co

ntri

bute

to th

is g

oal (

i.e.,

invo

lve

fam

ilies

,im

prov

e lin

kage

s)


1. Be child-focused and family-centred (i.e., family as partners, share information, strengths-based, respectful, honour di-versity, recognize developmental needs, fl exible services, culturally sensitive, and responsive);

2. Recognize the unique health needs of Aboriginal families;3. Support and value all service providers; and4. Focus on broader determinants of health.

Additionally, the Implementation Evaluation Framework was de-signed in relation to the fi ve goals of SACYHN:

1. Develop a shared vision and purpose;2. Identify and create opportunities for positive change in serv-

ice delivery;3. Involve families and service providers in signifi cant ways in

service planning and delivery;4. Build and sustain inter-regional, cross-sector, and clinical

connections; and5. Develop or improve electronic linkages.

The framework identifi ed both activity/process and health outcome indicators for each goal, and articulated how each outcome could be achieved through SACYHN’s committees and working groups. It also outlined valid and reliable data sources for each goal-specifi c outcome in order to facilitate meaningful interpretation of results.

Applying the Implementation Evaluation Framework

In the fall of 2004, with the framework established, SACYHN undertook an evaluation of the network in regard to its fi rst two goals, namely:

1. Develop and adopt a shared vision and purpose, and 2. Identify and create opportunities for positive change in serv-

ice delivery.

Six steps were involved in the evaluation:

1. Document review: A standard matrix was developed to record information related to each indicator for each of the two goals. This matrix was systematically applied to all docu-ments, commencing with the minutes of the SACYHN Steer-ing Committee meetings from inception to December 2004.


The document review also targeted SACYHN’s Background and Communications and Working Group materials.

2. Member survey: A questionnaire was developed, pretested, and revised. Content included current SACYHN involve-ment, individual and organizational impact, and benefi ts to service delivery; satisfaction with involvement; and thoughts about SACYHN’s future. The questionnaire was circulated to all individuals the SACYHN staff considered members, regardless of role or tie strength.

3. Semi-structured interviews with key stakeholders: Eight key informants were interviewed to provide context to the docu-ment review and survey fi ndings, and to validate preliminary conclusions. Key informants included parent participants; representatives from multiple child-serving sectors, the net-work governance structure, and network working groups; and network staff.

4. Child and Youth Advisory Council focus group: The ques-tions for SACYHN’s youth participants were parallel to the member survey but more tailored and administered in a focus group format. Members of the Council ranged in age from 12 to 19 years and length of participation in the Council ranged from one month to over one year.

5. SACYHN Director input: Interviews with the Director of SACYHN were designed to again provide additional context and validation as well as to gain a unique perspective on the network and its development.

6. Investigator analysis: Results from all sources were cross-referenced, themes and congruencies related to the two SACYHN goals were identifi ed, and a fi nal report was pre-pared which included recommendations for future considera-tion or action.

Substantive Findings about SACYHN That Provide Lessons about Network Characteristics and Evaluation

Overall, the evaluation fi ndings indicated positive outcomes related to the orientation of SACYHN’s structure, processes, and initiatives with respect to its fi rst two goals. The evaluation report included recommendations for future consideration that could have a positive impact on SACYHN realizing its vision of providing high quality, coor-dinated care as close to home as possible, of building and maintaining professional linkages, and of increasing capacity across regions. Full results from the SACYHN evaluation have been reported elsewhere


(Dolinski, 2005); only results that illustrate issues related to network characteristics and evaluation are reported here. Comparisons are also made with the formative evaluation completed by Rose (2004).

The ability of evaluation to inform network practices is illustrated by changes in understanding of commitment by SACYHN members. Initially, the formative evaluation by Rose (2004) identifi ed diffi culties amongst members in achieving consensus on the level of formality appropriate for SACYHN membership. However, the subsequent evaluation found that consensus had been achieved, with an associ-ated increase in formal membership in 2004. This positive fi nding supports the utility of evaluation in directing the efforts of networks, ultimately contributing to the probability of sustainability given that high turnover in stakeholder representation can lead to network extinction (Fawcett et al., 2000; Fonner, 1998).

The nebulous nature of networks was previously discussed as a chal-lenge in assessing network effectiveness. In the formative evaluation, Rose (2004) questioned whether the goal of SACYHN was to create “an organization or centre of excellence, or a system of excellence brought about by individuals and groups interacting in new ways?” (p. 20). The recent evaluation found that SACYHN must encourage its membership to appreciate that there are multiple ways to instill or operationalize the SACYHN vision in their community, and there-fore generate awareness about the network and build community capacity.

The recent evaluation also found that some individual members re-main uncomfortable with the “nebulosity” of SACYHN. However, use of the word “nebulous” does not imply that SACYHN has no structure or decision-making process. Rather, the nebulosity of SACYHN is represented by the confl ict or incongruence between the network’s desire to support more fl uid processes and individual expectations and experience of participation. From a network vision perspective, the drive to eliminate physical and psychological rigidity, in order to create an environment where ideas and information can circu-late freely, may be easily adopted by members. However, when this requires personal or organizational change or fl exibility (that is, challenges to traditional ways of working), members may experience considerable discomfort and the tendency can be to revert to creating structure to manage the anxiety. Many members remain confl icted between wanting/needing more structure, and wanting to be open to participating in what was described by an interviewee as “having


the conversation.” SACYHN’s literature review on networks identi-fi ed the “tyranny of structurelessness” (Gilchrist, 1995) as an issue that may lead to network disbandment. Therefore, SACYHN needs to continue to provide enough structure to offset member anxiety while still challenging traditional organizational frameworks in its pursuit to effect change.

A further question posed by Rose (2004) in the formative evaluation was whether the structure of SACYHN could sustain more work and, if so, what kinds of work would meet expectations for effectiveness/success in the long term? This question is directly related to the citation of role confusion as a challenge to assessing network effectiveness. However, too much activity can also be a pitfall for networks if mem-bers fall into a trap of engaging in activities that “give the illusion of accomplishment but are not appropriate for the network” (Chisholm, 1998, p. 194). As SACYHN currently identifi es priority areas of interest and action using stringent criteria, the potential for what Chisholm describes is signifi cantly reduced. Still, the recent evalua-tion detected some ambivalence around what work was being done, and what work should be done. Thirty-eight percent of questionnaire respondents felt that over the next fi ve years SACYHN should review how it decides upon its areas of activities.

Furthermore, some SACYHN respondents/informants wanted to focus on achieving more tangible outcomes, and in realizing these outcomes felt a need for them to be directly attributable to the network itself. In SACYHN, as well as in other collaborative partnerships, there is a need to link network successes back to participating organizations in order to demonstrate the value of the network to its participants (Taylor-Powell et al., 1998). Given the nature of networks described above, this direct attribution of outcomes will be extremely diffi cult to realize. Numerous members questioned whether it is even appro-priate for concrete outcomes to be linked back to SACYHN, as this is incongruent with the rationale and ethos of networks.

Connectedness was also cited as a challenge to the assessment of network effectiveness. Rose (2004) stated that “Networking (building relationships) requires a signifi cant amount of time and other invest-ment at the same time that individuals and organizations are feeling a squeeze on their time” (p. 20). Despite this sentiment, the majority of respondents acknowledged and appreciated the fact that trust bonds, and therefore connectedness, were developing well amongst SACYHN participants. It was critically important that the effective-


ness criteria in the Implementation Evaluation Framework enabled this fi nding, given that a lack of trust and commitment between and by network members has been attributed to network extinction (Child & Faulkner, 1998).

Lessons About Tools and Methods for Network Evaluation

The discussion below pinpoints a number of additional challenges and lessons related to the use of evaluation tools and methods in as-sessing network effectiveness, many of which can again be directly linked to the network literature.

Combining qualitative and quantitative data collection allowed for improved understanding of network impact by providing an in-depth and contextual description of how SACYHN was positioned both with its member base, and in the community. Goal 1 was measured against effectiveness criteria that encompassed “participant satisfaction” to “increased linkages with other child and youth health networks.” Goal 2 was measured against a broad range of effectiveness criteria that were not simply activity-focused. That is, as SACYHN (and many child and youth health networks in Canada) subscribe to a broad defi nition of service delivery, a purely quantitative investiga-tion of this goal would not have captured the entirety of initiatives. This fi nding is in keeping with recommendations in the literature for strengthening evaluation of collaborative partnerships through use of multiple designs (Taylor-Powell et al., 1998).

Generalized network member surveys should probably not be the dominant form of data collection. In our case, the use of a general e-mail member survey was not the most effective method. A response rate of 28% to the survey left questions about the extent to which the fi ndings were representative of all members. Anecdotal comments obtained from non-responders suggest that the questionnaire was too detailed in relation to the level of connection for many, and that some participants would have preferred to complete the questionnaire during scheduled committee meetings. Response rates in similar circumstances have ranged from 17% (Child Health Network for the Greater Toronto Area [CHN], 2003) to 48% (Child and Youth Health Network for Eastern Ontario [CHEO], 2000).

Despite its limitations, the SACYHN member survey did reveal es-sential information about member connectedness. Some members who completed the survey viewed themselves as strongly connected, while


others did not see themselves as formally connected. These fi ndings were of value in determining the extent of fragmentation amongst members, and provided insight into the distribution of strong versus weak ties throughout the network. Such information is critical for network development, as weak ties, once identifi ed, are generally easy to strengthen with targeted attention (Provan et al., 2005).

Differences in opinion as to the role of a network also arose as a challenge in SACYHN’s evaluation. Specifi cally, responses on the member survey as to what criteria should be used to assess network effectiveness may have differed depending on the respondent’s view of the primary role of SACYHN; yet the survey did not enable re-spondents to clearly articulate just what they consider the work of SACYHN to be. For example, the majority (in line with SACYHN’s vision) indicated an effectiveness criterion was “accessibility of child and youth health services” (87.5%), whereas for some it was their organization realizing “cost effectiveness” from participation (39%). It can be assumed that these differences would infl uence how par-ticipants responded to other questions related to how well SACYHN functions (Dolinski, 2005).

If surveys are used, some preliminary work on individual perceptions of membership ought to be done, as described in CHEO (2000), par-ticularly given the diversity of members and their varying connected-ness and ideas about network roles. Furthermore, network member surveys should be customized to each type of member role and perhaps to level of commitment or activity. As well, consideration should be given to using natural opportunities (i.e., scheduled meetings) to al-low completion of the survey. Another option for networks might be to consider using a social network analysis tool early on, since that method is designed to communicate to stakeholders (visually) where the relationships are, where they are not, where connections can be leveraged, and so on (Hawe, Webster, & Shiell, 2004; Provan, Veazie, Teufel-Shone, & Huddleston, 2004). This might afford later compari-sons with the increasing multiplexity, or connection of participants in more than one way, of the network over time as well as capture changing perspectives of participants.

The number of indicators should be limited, or better defi ned between goals, so that results can be accurately attributed when reporting against them. One of the SACYHN evaluation recommendations in-cluded review and revision of the indicators contained in the Imple-mentation Evaluation Framework (Dolinski, 2005). In an attempt to


be comprehensive, the framework identifi ed multiple indicators at varying levels, some of which were duplicated, ultimately making it diffi cult to operationalize the evaluation. Furthermore, to reduce both investigator and participant burden, it may be prudent to apply a time frame for measurement to each indicator. This would enable an im-proved focus on new initiatives as they develop, and provide a clearer perspective on network development and growth at any given time.

Triangulation of multiple sources of qualitative data assists with understanding network effectiveness. Completion of a SACYHN docu-ment review provided a perspective on how SACYHN was positioned. Qualitative data gleaned from the participant survey, semi-structured interviews, and focus groups added context to the data from the docu-ment matrix. The qualitative data clarifi ed what “high quality” means to SACYHN participants when one considers programs, services, and information resources. The recent evaluation of SACYHN’s fi rst two goals provided valuable information regarding actual commitment to vision and purpose as well as the impact on service delivery.

CONCLUSION AND FUTURE DIRECTIONS

Considerations for the Next Phase of SACYHN Evaluation

There were several challenges to applying what may be described as program evaluation methods and tools to evaluating our network. However, much was learned and SACYHN is now well positioned to plan and develop innovative evaluation tools and methodologies for the next stage of evaluation in order to capture the effectiveness of the complex inter-organizational relationships involved. Developing these innovative methods may be made easier by focusing on what now needs to be measured as an indicator of network effectiveness. That is, does SACYHN, now four years old, need to move away from evolutionary dynamics to focus on the impact on health outcomes? Or is it still too early to be measuring such outcomes, given that much of what SACYHN does is related to and designed to impact (but one step removed from) the service delivery that will encourage an improvement in, or degradation of, health outcomes? That is, what can be properly attributed to SACYHN initiatives and coordination? Discussion of these issues will assist SACYHN in developing goals that address the health outcome component of its mission.

With respect to methods, SACYHN now needs to consider whether it will narrow the scope of evaluation to focus less on the entire member


base, and more on recording and analyzing the development and out-comes related to particular initiatives, or to subgroups of participants. The focus will determine whether other approaches such as case studies, participatory action research, and/or social network analysis might provide valuable information about network effectiveness.

Considerations for Future Research on Evaluation of Networks

Evaluation is an issue of concern to child and youth health networks in Canada and, given the proliferation of networks, there is an ur-gent need to better understand network effectiveness, determine the indicators of a healthy or successful network, and develop net-work-specifi c evaluation methods (SACYHN, 2004). In the absence of thorough evaluation of network practices the progress of networks will be very diffi cult to measure, affecting the credibility of the net-work movement. SACYHN’s literature review (Hill, 2002) on network concepts and evaluation identifi ed areas for further research related to evaluating network effectiveness, including a need for:

• evaluations regarding the relationship between inter-organi-zational network structures and network effectiveness;

• evaluations that consider several levels of network opera-tional and impact analysis;

• evidence of impact at the community level;• evaluation of a network’s internal organizations;• consideration of both external and internal impacts of net-

works in evaluation; and • more description of networks as distinct from less complex

partnerships or community collaborative initiatives.

There is current deliberation over the potential to develop a multi-site network evaluation strategy across Canada that would provide child and youth health networks with common indicators of effec-tiveness. Individual networks are supportive of this direction, with 66.7% of respondents in the National Network Survey (SACYHN, 2004) indicating that it is very important to study network effective-ness. Furthermore, as evaluation mechanisms are rarely funded and developing outcome indicators for networks is a diffi cult process (Provan & Milward, 2001), there is value in working together so that all networks benefi t.


SUMMARY

This article presents key fi ndings from the literature relevant to network concepts and evaluation, and highlights learning from the evaluation activities of the Southern Alberta Child and Youth Health Network. Key lessons for evaluators examining network effectiveness include: consider a matrix approach that includes network operational and impact levels of analysis in evaluation; be thoughtful about the issue of membership connectedness when assessing effectiveness; carefully consider how well program evaluation methods and tools fi t in the more complex network context; and consider what outcomes can realistically and appropriately be tied directly to a network.

REFERENCES

Bolland, J.M., & Wilson, J.V. (1994). Three faces of integrative coordina-tion: A model of inter-organizational relations in health and human services. Health Services Research, 29, 341–366.

Child and Youth Health Network for Eastern Ontario (CHEO). (2000). Final report on evaluation from the Evaluation Task Group to the Network Steering Committee. Ottawa: Author.

Child and Youth Health Networks. (2003, June). Think tank on a national child and youth health network study: A dialogue on a scope and strategy for a national multi-site evaluation framework. Proceedings from a satellite event of the 2003 Canadian Association of Paediatric Health Centres (CAPHC) Conference, Calgary, AB.

Child Health Network for the Greater Toronto Area (CHN). (2003). Taking a fi rst step: Evaluating system performance for mothers, newborns, children and youth. Toronto: Author.

Child, J., & Faulkner, D. (1998). Strategies of cooperation: Managing al-liances, networks and joint ventures. New York: Oxford University Press.

Chisholm, R.F. (1996). On the meaning of networks. Group and Organiza-tion Management, 21, 216–236.

Chisholm, R.F. (1998). Developing network organizations: Learning from practice and theory. Reading, MA: Addison Wesley Longman.


Dockery, G. (1996). Rhetoric or reality? Participatory research in the NHS. In K. deKoning & M. Martin (Eds.), Participatory research in health: Issues and experiences (pp. 164–176). London: Zed Books.

Dolinski, C. (2005). Southern Alberta Child and Youth Health Network: Evaluation report. Calgary, AB: Southern Alberta Child and Youth Health Network.

Fawcett, S.B., Francisco, V.T., Paine-Andrews, A., & Schultz, J.A. (2000). A model memorandum of collaboration: A proposal. Public Health Reports, 115(2–3), 174–179.

Fonner, E., Jr. (1998). Relationship management: Assessing coalitions, networks, and other workgroups. Administrative Radiology Journal, 17(6), 18–21.

Gilchrist, A. (1995). Community development and networking. London: Com-munity Development Foundation.

Goss, S. (2001). Making local governance work: Networks, relationships and management of change. New York: Palgrave.

Gray, B., & Wood, D.J. (1991). Collaborative alliances: Moving from practice to theory. Journal of Applied Behavioral Science, 27, 3–22.

Hawe, P., Webster, C., & Shiell, A. (2004). A glossary of terms for navigat-ing the fi eld of social network analysis. Journal of Epidemiology and Community Health, 58, 971–975.

Hill, C. (2002). Network literature review: Conceptualizing and evaluating networks. Calgary, AB: Southern Alberta Child and Youth Health Network.

Lehman, A.F., Postrado, L.T., Roth, D., McNary, S.W., & Goldman, H.H. (1994). Continuity of care and client outcomes in the Robert Wood Johnson Foundation program on chronic mental illness. Milbank Quarterly, 72, 105–122.

Mays, G.P., Halverson, P.K., & Kaluzny, A.D. (1998). Collaboration to improve community health: Trends and alternative models. Joint Commission on Quality Improvement, 24(10), 116–123.


O’Toole, L.J. (1997). Treating networks seriously: Practical and research-based agendas in public administration. Public Administration Re-view, 57, 45–52.

Popp, J., Douglas-England, K., Casebeer, A., & Tough, S. (2005). Creating frameworks for providing services closer to home in the context of a network. Healthcare Management FORUM, 18(2), 27–33.

Provan, K.G., & Milward, B. (2001). Do networks really work? A framework for evaluating public-sector organizational networks. Public Admin-istration Review, 61, 414–424.

Provan, K.G., Veazie, M.A., Staten, L.K., & Teufel-Shone, N.I. (2005.) The use of network analysis for strengthening community partnerships in health and human services. Public Administration Review, 65, 603–613.

Provan, K.G., Veazie, M.A., Teufel-Shone, N.I., & Huddleston, C. (2004). Network analysis as a tool for assessing and building community capacity for provision of chronic disease services. Health Promotion Practice, 5, 174–181.

Rose, T. (2004). Research project report: Evolution, development, and main-tenance of the Southern Alberta Child and Youth Health Network. Calgary, AB: Southern Alberta Child and Youth Health Network.

Southern Alberta Child and Youth Health Network (SACYHN). (2004). Child and youth health networks: A national survey. Calgary, AB: Author.

Taylor-Powell, E., Rossing, B., & Geran, J. (1998). Evaluating collaboratives: Reaching the potential (Document G3658-8). Madison, WI: University of Wisconsin-Extension, Cooperative Extension.

Weiss, E.S., Anderson, R.M., & Lasker, R.D. (2002). Making the most of col-laboration: Exploring the relationship between partnership synergy and partnership functioning. Health Education and Behavior, 29, 683–698.

World Health Organization (WHO). (1998). Health promotion glossary. Di-vision of Health Promotion, Education and Communications (HPR), Health Education and Health Promotion Unit (HEP). (Document WHO/HPR/HEP/98.1). Geneva: Author.


Janice K. Popp, M.S.W., R.S.W., is a clinical social worker and Director of the Southern Alberta Child and Youth Health Network (SACYHN), an inter-regional, cross-sector health network for children and youth. Since SACYHN’s inception, Janice has had responsibility for the ongoing development and evaluation of the network and has a keen interest in connecting network theory to practice. She also has administrative responsibility for family-centred care, the Fam-ily and Community Resource Centre, and Palliative Care and Grief Support Services within the Child and Women’s Health portfolio in the Calgary Health Region. Previously, Janice worked in the area of children’s mental health as a therapist, program manager and in provincial policy and planning, all roles entailing extensive cross-sector collaboration.

Laura N. L’Heureux, M.A., is the Evaluation Coordinator for the Southern Alberta Child and Youth Health Network; the Principal and Owner of Panoptik Research and Consulting in Calgary, Alberta; and has served as the Vice-President of the Alberta Chapter of the Cana-dian Evaluation Society since 2003. Special research interests include the application of systems theory to evaluation, network evaluation, sustainable development, practice and policy outcomes related to health research, organizational effectiveness, harm reduction, and access to health care. She obtained her interdisciplinary graduate training in program evaluation and applied research from the Applied Social Psychology Department at the University of Saskatchewan.

Carly M. Dolinski, M.P.H., received her master’s degree from the Department of Population Health, University of Western Australia, where her research explored the health of child care workers from a policy perspective. She has worked as a clinical nurse, project co-or-dinator, evaluator, and research assistant, with particular interests in remote health care, Aboriginal health, and maternal and child health. Most recently, she completed a research assistant post with the Calgary Health Region’s Decision Support Research Team based at the Alberta Children’s Hospital. Now back in Australia, Carly will soon be commencing a new position as a program offi cer with the Strategic Planning Branch of the Offi ce of Mental Health Division at the Health Department of Western Australia.

Carol E. Adair, M.Sc., Ph.D., is Associate Professor in the Depart-ments of Psychiatry and Community Health Sciences and co-lead of the Population Mental Health Research Program in the Hotchkiss Brain Institute at the University of Calgary. Her research interests


around health services include evaluation/outcomes research and paediatric and psychiatric epidemiology.

Suzanne C. Tough, Ph.D., is Associate Professor in the Departments of Paediatrics and Community Health Sciences as well as Associate Director for the Institute of Maternal and Child Health at the Uni-versity of Calgary. She is also the Scientifi c Director of the Alberta Centre for Child, Family and Community Research and an Alberta Heritage Foundation for Medical Research (AHFMR) Population Health Investigator. Her research program focuses on the health and well-being of mothers and infants beginning prior to conception through infancy to optimize birth and childhood outcomes.

Ann L. Casebeer, Ph.D., is Associate Professor in the Department of Community Health Sciences at the University of Calgary. Her research efforts focus on understanding change supporting health improvement, with special interests in organizational innovation in the context of health systems, advancing policy incentives, and enhancing public participation in decision making for health and health care. As Faculty Director for SEARCH Canada (Swift Effi cient Application of Research in Community Health; a public service or-ganization dedicated to improving knowledge development and use in practice), Ann is particularly interested in supporting learning for innovation and in sharing and using learning across challenging boundaries and borders.

Kathleen L. Douglas-England, M.Sc., is an adjunct professor in the Department of Paediatrics, University of Calgary, and a Senior Research Consultant in the Quality, Safety and Health Information Portfolio, Calgary Health Region. Her program of research and train-ing includes survey instrument development, program evaluation for health services, and expertise in patient and family feedback about the health care experience.

Catherine C. Morrison, M.S.W., is the Calgary Region Network Manager for the Southern Alberta Child and Youth Health Net-work. She has over 20 years of experience within the health and children’s services sectors. Her graduate work focused on networks, their functioning and evaluation, and she has an ongoing interest in strengthening connectivity between organizations that serve children and their families.

how do you evaluate a network? a canadian child and …

Documents