using standing setting in evaluation: exploring differences between the ordinary and excellent
Post on 25-Feb-2016
25 Views
Preview:
DESCRIPTION
TRANSCRIPT
Using Standing Setting in evaluation: Exploring
differences between the ordinary and excellent
Janet Clinton University of Auckland, New Zealand
University of Connecticut 2009
The chocolate chip cookie exercise
Chocolate chip standards of success
Characteristic
Poor Basic Good Excellent
Chewy centreHealth factor
Number of chipsSize
The x factor
Aims
Explore judgment of successIllustrate standard setting as an approach
to systematically judge levels of successDemonstrate that standard setting can be
a powerful tool in program evaluation in either large or small evaluations
Standard setting particularly useful in health settings
What is a standard?
A standard is a statement about
whether a performance is good
enough for a particular purpose
Evaluation is about judgments
The judges
How have we done this in the past?
The use of benchmarks to understand success:
Common practice Developed from literature reviews &
input by stakeholders however Often inaccurate or not representative
The lack of strong evidence and diverse cultures often makes the process of determining standards difficult
Praying……….
What is standard setting?
Standard setting is a means of systematically
harnessing human judgment to define and
defend a set of standards
Standard setting methodology
Describes the attributes of each category e.g., what makes a program fall into the category of excellent
A process of rationally deriving, consistently applying, and describing procedures on which judgments can be made
The process decides on a “cut-score” that then creates distinct and defensible categories e.g., pass/fail, allow/deny, excellent/poor.
Theoretical basis & method
Based on theories of judgment analysis & attribution theory
Used in educational measurement e.g. Jaegar, 1997
Methods◦ Angoff◦ Bookmark◦ Hofstee◦ Lens Modelling
The method has to be:DefensibleCredible Supported by body of
evidence in the literature
Feasible Acceptable to all
stakeholders
All about the process and the right fit
Standard Setting in an Evaluation of
National Chronic Care Management
Initiatives
The project
Understand best practice of chronic care management programs --- nationally.
Develop a user friendly best practice workbook: e.g. COPD, Stroke, CVD & CHF
◦ The evidence---Literature review◦ What is happening---Stock take survey◦ Key stakeholders--Expert interviews◦ Acceptable practice--Standard setting exercise◦ Evaluation of individual programs or sites◦ Development of a best practice work book
Applying the methodology
Conducted Stand Setting Workshops
◦5 sites around NZ ◦Invite experts/stakeholders◦Group and individual rating exercises ◦Analyze the assessments◦Define a standard ◦Determine success indicators
Preparation work
1. Understand the evidence base for success
2. Develop a set of dimensions 3. Develop profiles of programs
4. Plan the workshops
5. Determine the composition of stakeholders
6. Develop assessment rubrics—scoring schedule
Exercise 1
Importance of each factor ◦Rank the importance of the factors
that make up best practice◦Present the information back◦Discuss as a group e.g. missing
dimensions
DIMENSION EXPLANATION
LEVEL OF SUPPORT FOR CHRONIC CARE
Limited Basic Reasonably Good Fully Developed 1 2 3 4 5 6 7 8 9 10
Conceptual understanding of CCM
Appropriate levels of collaboration
Active engagement of leadership
Development of sustainable community links
Focus on health (inequalities)
Decision support systems in place
Appropriate delivery design system
Knowledge transfer
Attention to efficiency/ cost/ output
Attention to effectiveness/ outcomes
Adherence to clinical guidelines
Overall perception of the programme
ID CODE: PROFILE NAME:
Site W
0
2
4
6
8
10
12
Con
cept
ual
unde
rsta
ndin
g of
CC
MPa
tient
Appr
opria
te le
vels
of c
olla
bora
tion
Activ
e en
gage
men
tof
lead
ersh
ip
Appr
opria
tede
velo
pmen
t of
sust
aina
ble
com
mun
ity lin
ks
Focu
s on
hea
lth(in
equa
litie
s)
Dec
isio
n su
ppor
tsy
stem
s in
pla
ce
Appr
opria
te d
eliv
ery
desi
gn s
yste
m
Know
ledg
e tra
nsfe
r
Atte
ntio
n to
effic
ienc
y / c
ost /
outp
ut
Atte
ntio
n to
effe
ctiv
enes
s /
outc
omes
Adhe
renc
e to
clin
ical
gui
delin
es
Ave
rage
Rat
ing
(+1
SD)
Regional Comparison
0
1
2
3
4
5
6
7
8
9
10
11C
once
ptua
lun
ders
tand
ing
ofC
CM
Appr
opria
te le
vels
of c
olla
bora
tion
Activ
e en
gage
men
tof
lead
ersh
ip
Appr
opria
tede
velo
pmen
t of
sust
aina
ble
Focu
s on
hea
lth(in
equa
lities
)
Dec
isio
n su
ppor
tsy
stem
s in
pla
ce
Appr
opria
te d
eliv
ery
desi
gn s
yste
m
Know
ledg
e tra
nsfe
r
Atte
ntio
n to
effic
ienc
y / c
ost /
outp
ut
Atte
ntio
n to
effe
ctiv
enes
s /
outc
omes
Adhe
renc
e to
clin
ical
gui
delin
es
Ave
rage
Rat
ing Rotorua
WhangareiChristchurchAucklandWellington
Regions
The Exercise 2: Round 1
• Form representative groups • Describe and discuss the dimension• Read the profiles & make a judgment about
the standard of the dimensions• Rate the dimensions• Each individual records the reason or
explanation for their choice
DIMENSION EXPLANATION
LEVEL OF SUPPORT FOR CHRONIC CARE
Limited Basic Reasonably Good Fully Developed 1 2 3 4 5 6 7 8 9 10
Conceptual understanding of CCM
Appropriate levels of collaboration
Active engagement of leadership
Development of sustainable community links
Focus on health (inequalities)
Decision support systems in place
Appropriate delivery design system
Knowledge transfer
Attention to efficiency/ cost/ output
Attention to effectiveness/ outcomes
Adherence to clinical guidelines
Overall perception of the programme
ID CODE: PROFILE NAME:
Analyzing the information
Analyze all the information
◦Present the information back to the established groups of mixed stakeholders
◦Individual information was returned
Exercise 2: Round 2
Information returnedGroups discuss their individual
choicesAim to get group consensus on
rating best practice of dimensionsInformation recorded along with
the explanations
DHB1 Self Perceptions - Average Ratings for Dimensions
0123456789
1011
Empo
wer
men
tan
d Se
lf-M
anag
emen
t
Collabo
ratio
n
Lead
ersh
ip
Com
mun
ityLink
ages
Red
ucing
Ineq
ualiti
es
Dec
ision
Supp
ort
Deliver
y:or
ganisa
tion
of h
ealth
care
Deliver
y:pr
ogra
mm
ede
liver
y
Know
ledg
eTr
ansf
er
Effic
ienc
y
Effe
ctiven
ess
Clin
ical
Guide
lines
Roles defined,
attendance at meetings;
program spokes person
No collaborati
on with PHO or NGOs
No plan to disseminate info to community
Results for each profile
Analysis of results
Regression to determine importance of dimensions
Create a matrix of explanations first by individuals then by group consensus
Consult the evidence & the experts Determine the ‘cut scores’ for each
assessment area—used the bookmark method
N Mean sd
Engagement of leadership 479 5.23 1.81Focus on health (inequalities) 479 4.97 2.09Collaboration 478 4.96 1.86Adherence to clinical guidelines 477 4.88 1.95Conceptual understanding 475 4.76 2.00Community links 475 4.67 1.86Attention to effectiveness / outcomes 476 4.65 2.03Attention to efficiency / cost / output 468 4.63 2.01Delivery design system 469 4.40 1.70Decision support systems 478 4.39 1.79Knowledge transfer 476 4.21 1.81
Overall 480 4.72 1.76
Overall means
Factor analysis Individuals Groups
Program & Organization 1 2 3 1 2 3Community links .92 .06 -.18 .69 .06 .06Collaboration .74 .04 .05 .90 -.00 -.07Focus on health inequalities .72 .01 .00 .75 -.10 .17Conceptual understanding .61 -.07 .21 .86 .06 -.08Delivery design system .54 .05 .22 .52 .36 -.01Engagement of leadership .40 .06 .39 .48 .24 .24
Effectiveness/EfficiencyAttention to efficiency -.03 .97 -.03 .04 .91 -.02Attention to effectiveness .10 .73 .12 .15 .79 -.03
InformationDecision support systems .08 .10 .72 .30 .19 .70Adhere to clinical guidelines .05 .21 .59 -.11 .67 .32Knowledge transfer .14 .36 .39 .43 .48 -.00
Factor inter correlationsProgram 1 1Efficiency/Effectiveness .62 1 .68 1Information .66 .68 1 .38 .51 1
Across the exemplars
1 5 10 3 8 6 7 2 4 9 Total1.00
2.00
3.00
4.00
5.00
6.00
7.00
Grp- Final 4 Ind - Final 4 Gp - All Ind - All
Reasonably good
Fully Developed
Limited
Basic
DIMENSION
LEVEL OF SUPPORT FOR CHRONIC CARE
LIMITED BASIC REASONABLYGOOD
FULLY DEVELOPED
1 2 3 4 5 6 7 8 9 10
CONCEPTUAL UNDERSTANDING
COLLABORATIONACTIVE LEADERSHIPCOMMUNITY LINKSFOCUS ON INEQUALITIESDECISION SUPPORTDELIVERY SYSTEMKNOWLEDGE TRANSFER
EFFICIENCY COST/ OUTPUT
EFFECTIVENESS-OUTCOMES
USE OF CLINICAL GUIDELINES
OVERALL
The standard
Dimension Limited Basic Reasonably good
Excellent
Collaboration No work with communityNo evidenceHospital focusPoor referral system
Low engagement with Maori low levels of trustLittle primary/ secondaryintegration
Recognises weaknessesEvidence of partnershipsGood initiatives
Community approach evidencedWhole system collaborationLots of alternatives
Leadership No championPoor managementLack of evidence
Some leadershipNo championsFoundation of leadership
Strong clinical leadershipWeak championsIdentified problems but no change
Evident leadership at programStrong championsEvidence of vision
Did it work?
Analysis illustrates that we can create cut scores for basic through to excellent.
We can define the judgment Have an explanation or criteria for
each level
Reactions to the workshops
◦ We have a voice◦ Systematic◦ Inclusive for roles,
regions & view◦ Useful for self review
◦ Hard work◦ We got nothing◦ This is dangerous◦ Stupid!
Variable
Draw backs
Resource intenseAnalysis can be problematicTime PressuresSetting league tables in healthScary
Consequences
Creates a benchmark for judgmentValidates research through practiceUnderstand different views of successFacilitates self-reviewEncourages learningEncourages an evaluation perspective
Working with Community groups
Adapt the processNeed 8-10 in groupGreat focus of the explanation Qualitative analysisPreparation takes community into
accountUse of evidence to build rubrics &
exemplarsFeasibility-Propriety- Accuracy-Utility
Final word
Not so much◦the METHOD as the PROCESS
The judgment
top related