use of fidelity measurement in statewide technical assistance activities gary bond festschrift...
TRANSCRIPT
Use of Fidelity Measurement Use of Fidelity Measurement in Statewide Technical in Statewide Technical Assistance ActivitiesAssistance Activities
Gary Bond Festschrift ConferenceGary Bond Festschrift ConferenceIndianapolis, INIndianapolis, IN
September 24, 2009September 24, 2009
Angela L. Rollins, Ph.D.Angela L. Rollins, Ph.D.ACT Center of IndianaACT Center of Indiana
IUPUI Psychology DepartmentIUPUI Psychology Department
OverviewOverview
Background on fidelity use in Background on fidelity use in generalgeneral
Use for fidelity in technical Use for fidelity in technical assistance (TA) efforts across assistance (TA) efforts across various EBPsvarious EBPs
Some new directionsSome new directions Further questionsFurther questions
Uses of Fidelity: Uses of Fidelity: ResearchResearch
Internal validity: examine model Internal validity: examine model adherence/driftadherence/drift
Facilitating communication in literatureFacilitating communication in literature Synthesizing a body of researchSynthesizing a body of research Identifying critical ingredients of Identifying critical ingredients of
program modelsprogram models Defining model adaptationsDefining model adaptations
Uses of Fidelity: TAUses of Fidelity: TA Benchmarks to add perspective
Over timeOver time Comparison to other programsComparison to other programs Comparison to normsComparison to norms
Coaching Written report of strengths, weaknesses,
recommendations for improvement Teaching moments within the
assessment day itself
Uses of Fidelity: TAUses of Fidelity: TA
Defining program standards/critical ingredients
Monitoring adherence to program Monitoring adherence to program standards – tricky new areastandards – tricky new area
Springboard for other TA activitiesSpringboard for other TA activities Data gathering (Indiana)Data gathering (Indiana) Outcomes data reporting (Kansas)Outcomes data reporting (Kansas)
Indiana ACT Fidelity Indiana ACT Fidelity ApproachApproach
2 assessors 2 assessors 1 1 Semi-annual Semi-annual annual after first annual after first
yearyear Even with these reductions, as # Even with these reductions, as #
teams grow and # consultants teams grow and # consultants remains constant, fidelity visits take remains constant, fidelity visits take a bigger piece of the overall a bigger piece of the overall consulting pieconsulting pie
Indiana ACT Fidelity Indiana ACT Fidelity ApproachApproach
Testing phone fidelity measures Testing phone fidelity measures (see (see McGrew talk tomorrow)McGrew talk tomorrow)
Level of burden has reduced a little over Level of burden has reduced a little over time, even for onsite assessmenttime, even for onsite assessment Team leaders more familiar with requestsTeam leaders more familiar with requests Agencies create reports over time (trust?)Agencies create reports over time (trust?) Assessors more adept as well - completing faster Assessors more adept as well - completing faster
than in initial yearsthan in initial years
Still need ongoing support - reduce rater Still need ongoing support - reduce rater drift, feedback on scoring decisionsdrift, feedback on scoring decisions
Multiple Uses of the VisitMultiple Uses of the Visit Data gatheringData gathering
Admission and discharge data for trackingAdmission and discharge data for tracking Tracking raw data used to compute fidelity Tracking raw data used to compute fidelity
scoresscores raw turnover ratesraw turnover rates actual FTE and caseload sizeactual FTE and caseload size actual frequency and intensityactual frequency and intensity
Teaching moments: most extensive Teaching moments: most extensive contact with site for the year in most contact with site for the year in most casescases
Multiple Uses of the VisitMultiple Uses of the Visit Report team’s scores over time in each Report team’s scores over time in each
report (examine annual progress)report (examine annual progress) Additional monitoring of state standards Additional monitoring of state standards
compliancecompliance Additional information added to report where Additional information added to report where
DACTS and state standards differDACTS and state standards differ Noncompliance reported to team initially, to Noncompliance reported to team initially, to
state if not remediated within ~90 daysstate if not remediated within ~90 days Research on real-world data, but problems Research on real-world data, but problems
with restriction in range (state standards with restriction in range (state standards dictate high fidelity)dictate high fidelity)
Indiana DACTS dataIndiana DACTS dataHuman Res Organizati
onServices DACTS Total
Baseline n=34
4.05 (.50)
4.14 (.56)
3.33 (.60)
3.81 (.43)
12 mos n=33
4.28 (.34)
4.56 (.42)
3.70 (.52)
4.14 (.34)
24 mos n=30
4.24 (.27)
4.68 (.23)
3.80 (.44)
4.20 (.26)
36 mos n=27
4.28 (.30)
4.69 (.19)
3.80 (.35)
4.21 (.20)
48 mos n=19
4.24 (.34)
4.65 (.18)
3.97 (.29)
4.25 (.20)
60 mos n=13
4.33 (.34)
4.63 (.26)
3.82 (.58)
4.22 (.34)
72 mos n=13
4.24 (.34)
4.67 (.11)
3.73 (.29)
4.17 (.17)
84 mos n=2
4.41 (.32)
4.86 (.00)
3.95 (.21)
4.36 (.20)
Fidelity Monitoring: IMRFidelity Monitoring: IMR Semi-annual visitsSemi-annual visits ““Baseline” after 3-6 months is more meaningful for Baseline” after 3-6 months is more meaningful for
program developmentprogram development Emphasis on qualitative feedback rather than Emphasis on qualitative feedback rather than
numeric scorenumeric score Data collection via observation of IMR sessions is Data collection via observation of IMR sessions is
vitalvital Immediate feedback at the end of visit important Immediate feedback at the end of visit important
for QI function of fidelityfor QI function of fidelity In progress: fidelity via audiotaped sessions – In progress: fidelity via audiotaped sessions –
based on Mueser’s workbased on Mueser’s work
Across EBPsAcross EBPs
Role of fidelity may vary in emphasis Role of fidelity may vary in emphasis across EBPsacross EBPs Example: Does fidelity measurement of Example: Does fidelity measurement of
IMR play as large a role in IMR play as large a role in implementation as it does for ACT?implementation as it does for ACT? Clinical competency needs pronounced for Clinical competency needs pronounced for
IMR, IDDT (see Mueser talk tomorrow)IMR, IDDT (see Mueser talk tomorrow) Program structural needs for ACT very Program structural needs for ACT very
pronouncedpronounced
Future DirectionsFuture Directions Increasing calls to decrease the burden Increasing calls to decrease the burden
of fidelity measurement (e.g., We want of fidelity measurement (e.g., We want to use one FTE or less for monitoring to use one FTE or less for monitoring our whole state)our whole state) Phone/remote, less intensive version (see Phone/remote, less intensive version (see
McGrew)McGrew) Lower frequency of assessmentsLower frequency of assessments Screening approachScreening approach
key itemskey items follow-up with full visit if flagged during screenfollow-up with full visit if flagged during screen
Future DirectionsFuture Directions Ohio developing The Evaluation Database: Ohio developing The Evaluation Database:
“TED”“TED” Online data entry by consultant after visitOnline data entry by consultant after visit
Perhaps during onsite visit with wireless cardPerhaps during onsite visit with wireless card More automated consensus and report functionMore automated consensus and report function Track when consultants enter data and when reports Track when consultants enter data and when reports
go out: QA for the TAgo out: QA for the TA Raw data elements as well for fuller data gatheringRaw data elements as well for fuller data gathering IDDT and (SE?) so far, may collaborate to tailor to IDDT and (SE?) so far, may collaborate to tailor to
ACT ACT Seeking funding to tie in consumer outcomes as wellSeeking funding to tie in consumer outcomes as well Contact: [email protected]: [email protected]
Questions to PonderQuestions to Ponder Are statewide fidelity scores a good Are statewide fidelity scores a good
measure of the TA center’s effectiveness?measure of the TA center’s effectiveness? Would outside fidelity assessors (outside Would outside fidelity assessors (outside
the state/entity) produce similar results?the state/entity) produce similar results? Beyond basic requirements for Beyond basic requirements for
certification, what motivates teams to score certification, what motivates teams to score high on fidelity scale?high on fidelity scale?
Can we marry research and practice when Can we marry research and practice when practice yields a restriction in range on practice yields a restriction in range on fidelity scores?fidelity scores?
Thank youThank you
Charles BoyleCharles Boyle Michelle SalyersMichelle Salyers Lia HicksLia Hicks Deb HroudaDeb Hrouda