program evaluation unit, univerity of melbourne evaluating organisational context measurement...

Program Evaluation Unit, Univerity of Melbourne

Evaluating organisational context

Measurement approaches: evaluation

David Dunt


The contribution of realistic evaluation in studying the effects of organisational context on KU & KU’s take-up at grassroots level


For me, Realistic evaluation, not a new set of ideas but a name for an approach that I had progressively discovered during my career as an evaluator (‘that’s what it is’)

My outline not so much an exposition on a realist theory of science

But a description of this path of discovery As it encapsualtes the path to

enlightenment for many other health practitioners interested in KU particularly after their discovery of Evidence-based Medicine & healthcare


Outline the main ideas of Realistic evaluation particularly the key concepts of CMOs Context

Mechanism

Outcomes Outline applications to KU &

organisational context


Outcomes movement- point of departure

The randomised control trial

TreatmentProgram

Outcome‘Thing’


Outcomes movement- first problem

The randomised control trial

TreatmentProgram Outcome‘Thing’

Treatment, program content - a black box


With a black box:-

Don’t know what element of the program was responsible for success failure

Was program failure due to model failure or implementation failure (of a good model)


Distinction is fundamental to evidence-based decision making where effective and ineffective programs must be correctly identified


Points to the importance of the study of process

The importance of the use of Program logic


Program logic Is a concept much loved by evaluators Similar to management flowsheets

Specify successive stages (chronological, causal) in program rollout

Use key performance indicators (KPIs) to assess whether a stage has been successfully reached

Gather relevant information collected at each stage why stage has been reached (or not)

But need to move beyond their common sense models of causation & chronology


Program theory - Mechanisms

Instead program logic based upon a program theory in order to identify optimal program content including all its components and their relationship (stages)

Application of a Mechanism or causal model based on best information (literature, experts, stakeholders)


Program theory - Context Program logic also based upon

knowledge of local Context (analysis of barriers, risks eg negotiating key gatekeepers, usual work patterns, usual organisational practice and barriers


In summary With program development

There should be an analysis of the application of the proposed Mechanism in the local Context in order to ensure a program with maximum functionality in that context and with achievement of best Outcomes

With program implementation There should be monitoring of the important

stages of the program’s rollout to ensure they are occurring


A second problem for the Outcomes movement embedded

in this analysis

Context is assumed and Context of EBM studies typically is

quite atypical Teaching hospitals Atypical patients chosen for their ease

of study (no co-morbidities, English-speaking, no mental impairment, and with unrepresentative social & demographic characteristics)


Fundamental to evidence-based decision making where effective and ineffective programs must be correctly identified


Realistic evaluation in KU

Focus on Context and Mechanism occurring to some extent already

EPOC group Examples: academic detailing,

use of opinion leaders


Realistic evaluation – in KU

Still little program theory (psychological, sociological [systems], or economic)

Confirmed in a recent UK review by Grimshaw et al

Reliance instead on atheoretical, tickbox questionnaire-based study of behavior change

Program logic approaches notably absent


Realistic evaluation - measuring organisational

context & change 1

Challenge is first to build program theory

Application of Mechanisms using existing theory and evidence to

Local organisational Contexts using

best local knowledge


Realistic evaluation - measuring organisational

context & change 2

Generally, aim is to build heuristic models to assist in thinking about program development and implementation.

Not to construct a causal model by eg path analysis, with each path’s contribution being measured by a correlation coefficient


A few exceptions where may be important to build causal

models

For very important and central dimensions of the health care system


A few exceptions where may be important to build causal

models

Finish briefly with an example of this from my own work in the Australian health care system

The development of a psychometric tool to measure the level of integration of the GP with the remainder of the health care system


GP integration with the health care system 1

70 question inventory completed by GPs about their current activities relevant to integration

9 integration factors organised around 2 higher order factors

patient care management and community health

Plus 5 enabling factors



Two key findings It is possible to map levels of GP

integration across Australia – considerable variation exists

Levels strongly relate to GP’s knowledge of local services and resources



Further details are a story for another day

program evaluation unit, univerity of melbourne evaluating organisational context measurement...

Documents

program evaluation unit

program failure

program rollout

program development

use of program logic

optimal program content

univerity of melbourne

evaluation david dunt