Australasian Evaluation Society ConferenceSydney August-September 2011

Strength of partnerships: a mixed methods approach to evaluation

Presented by:Date:

Geraldine Marsh

Virginia Lewis

Kate Silburn

Jenny Macmillan

Centre for Health Systems Development (CHSD) at the Australian Institute for Primary Care & Ageing (AIPCA) at La Trobe University has been evaluating initiatives that involve partnering at many levels.

Examples of the different types and levels of partnering initiatives we have evaluated include: Hospital Admission Risk Programs (HARP) Early Intervention for Chronic Disease projects Community Initiatives for National Suicide Prevention Strategy Two large-scale evaluations of statewide primary health system

development strategies in Australia: Primary Care Partnerships (Vic) and Connecting Healthcare in Communities (CHIC)

2

Role of Government

Governments have an important role in creating or enabling the foundations of effective systems for organised primary health care service delivery (Takach, Gauthier, Sims-Kastelein, & Kaye, 2010; World Health Organisation, 2008).

This requires deliberate and sophisticated policy processes that maximise the potential within complex systems and the capacity within these systems for innovation (McDaniel & Driebe, 2001; Swantson, et al., 2010; World Health Organisation, 2008).

Partnering and Primary Health Care Reform

Current reform is based on setting up (new) collaborative structures and relationships

Other geographically based formal and informal governance and organisational arrangements exist

Evaluative research is required to assess the emerging ML/LHN and their relationships to existing (or other developing) partnerships and the effects on PHC delivery and consumer outcomes

4

How do we assess value of partnering?

Evaluating partnering (collaboration, etc.) is important to demonstrate it adds value and is worth investment

Attributing the contribution of relationships between organisations to health service delivery and health outcomes is difficult

First step is being able to capture the nature of the partnership and describe its “effectiveness” in a systematic way

Measures

Literature on barriers and facilitators leads to an understanding of characteristics of functional partnerships, general consensus about what good partnerships look like. But how to capture systematically and usefully is difficult

Descriptive qualitative review Network mapping technique Questionnaires based on presence of characteristics

VicHealth Partnerships Analysis Tool Nuffield Partnership Assessment Tool (Hardy, Hudson and Waddington,

2003)

Commonly used questionnairesVicHealth Partnerships Analysis Tool

A tool for establishing, developing and maintaining productive partnerships

7 domains rated on a 5-point Agree/Disagree scale:1. Determining the need for the partnership2. Choosing partners3. Making sure partnerships work4. Planning collaborative action5. Implementing collaborative action6. Minimising the barriers to partnerships7. Reflecting on and continuing the partnership

Different possible methods of completion:Leaders present views for discussion; Individually completed; as a group activity

Nuffield Partnership Assessment Tool (Hardy, Hudson & Waddington, 2003)

Six domains rated on 4-point Agree-Disagree scale1. recognise and accept the need for partnership2. develop clarity and realism of purpose3. ensure commitment and ownership4. develop and maintain trust5. create clear and robust partnership working arrangements6. monitor, measure and learn

Completed individually with post-completion discussion

Potential issues

Each partner weights criteria for judging success differently (Toulemonde et al., 1998)

Sensitivity issues (e.g. Funnell, 2006; own experience leading to modified response options for original VicHealth tool)

Ratings tend to be “generous”

Discussion after completion of the tool is constrained by the ratings given

An alternative approach

Through extensive experience in evaluating partnerships CHSD has developed:

A mixed-method design for assessing partnerships in health systems

A generalised approach where focus of strategy/funding is on building capacity through inter-organisational (sector) processes

Procedure

Data collection involves a facilitated group interview ( partnership/governance group)

Questions are asked about the stage of development or progress: How well developed are each of the following in your Partnership Council?e.g. clear goals for PC

A facilitated discussion occurs

Two facilitators are present to allow post-completion review and moderation if required. Also enables accurate documentation of comments during discussion.

Questions (16) reflect domains represented in Nuffield and VicHealth

Includes references also to the underlying logic of the initiative/programIn the case of statewide primary health reform, some references to the health system within the catchment

Also collect a brief version of VicHealth Partnerships Analysis Tool (individually completed ) and a single overarching development question (consensus)

12

Response options relate to progress towards achievement of the characteristic:

We won’t be doing this Haven’t started yet Currently planning this Developing this This is reasonably well developed This is very well developed/ sustainable

No group consensus reached

Generality of the questions allows for flexibility and diversity in implementation of strategies, while facilitated discussion gathers specific information reflecting differences in local implementation the “why” is the same, the “where are we up to” is on the same scale, but

the “how” is different

Analysis of relationships between measures supports validity and reliability of the approach

15

Partnership Council ratings: “Clear goals for the PC ”

0

2

4

6

8

10

12

2008 (n=14) 2009 (n=14)

2011 (n=15)

Stages of Development

No.

of P

Cs

16

“Working so partners are involved in planning / setting priorities for collaborative action”

0

2

4

6

8

10

122008 (n=14)2009 (n=13)2011 (n=15)

Stages of Development

No. o

f PCs

Benefits of this Approach

Provides quantitative and qualitative data about the characteristics of the partnership

Immediate information is used by partnership to plan future actions – feedback supports strategic planning

Immediate information is used by funder/policy makers to identify areas where additional support and/or modification to strategy may be required

Qualitative information strengthens understanding and interpretation of quant data

17

Quantitative and qualitative data allows comparison of the characteristics of a partnership over time.

Contributes useful evidence to support development of the partnership and the policy/strategy

Data is about progress towards the strategic goal of increased system capacity through partnership.

Requires experienced and skilled facilitators – for the process of reaching consensus, and with ability to assess quality of evidence provided for ratings

Conclusion

The mixed method design can be applied to different settings to provide robust quantitative indicators with a rich level of interpretive information.

The data obtained through the facilitated group interview contributes useful evidence to support development of the partnership and the policy/strategy.

Associate Professor Virginia Lewis or Geraldine Marsh

Centre for Health Systems Development

Australian Institute for Primary Care & Ageing

La Trobe University

Email: v.lewis@latrobe.edu.au g.marsh@latrobe.edu.au

Contact:

Partnership Analysis Tool (Brief version – 20 items)

21

Determing the need for the partnership

Choosing partners

Making sure partnerships work

Planning and implementing collaborative action

Minimising the barriers to partnerships

Reflecting on and continuning the partnership

0

1

2

3

4

5

200820092011

1 = ‘Strongly disagree’2 = ‘Disagree’3 = ‘Not sure’4 = ‘Agree’ 5 = ‘Strongly agree’