evaluating a ryan white quality management program using quantitative and qualitative methods
DESCRIPTION
Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods. Ryan White All Grantee Meeting November 29 ,2012 Presenters: Mary Kay Diakite, LMSW and Graham Harriman, MA HIV Care, Treatment and Housing Program - PowerPoint PPT PresentationTRANSCRIPT
Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods
Ryan White All Grantee MeetingNovember 29,2012
Presenters: Mary Kay Diakite, LMSW and Graham Harriman, MAHIV Care, Treatment and Housing ProgramNYC Department of Health & Mental Hygiene
Disclosures• This continuing education activity is managed
and accredited by Professional Education Service Group. The information presented in this activity represents the opinion of the author(s) or faculty. Neither PESG, nor any accrediting organization endorses any commercial products displayed or mentioned in conjunction with this activity.
• Commercial Support was NOT received for this activity.
Disclosures•Mary Kay Diakite, LMSW
• Has no financial interest or relationships to disclose
•Graham Harriman, MA• Has no financial interest or relationships to
disclose
Learning ObjectivesAt the end of this activity, the participant will be able to:
1. Apply quantitative and qualitative methods in quality management program evaluations
2. Determine the most appropriate methods to apply in their particular EMA
Agenda•Background•Objectives•Methodology•Results
▫Quantitative▫Qualitative
•Discussion
Background• Quality management (QM) program goals
▫ Comprehensive performance measurement systems are developed & utilized to assess the quality of Part A services
▫ Enable Part A providers to continue to be skilled in using QI tools & methodologies to enhance the quality of their services
▫ Part A services improve health outcomes and quality of life for people living with HIV/AIDS
▫ The Part A QM Program remains responsive to the changing science & epidemiologic needs of the NY EMA community
Quality Management Activities
Organizational
Assessments
Performance Measuremen
t
Quality Management
Plans
Quality Learning Networks
Evaluation Objectives
Assess the Ryan White Part A QM Program conducted through the NYSDOH AIDS Institute during Jan-Dec 2011
Identify specific areas for improvement
Identify best practices & plan future trainings
Survey Overview
General perception
sAgenc
y
OALN
meetings
QI training
s
Performance
Indicators
Methodology & Timeline
Survey Development (Fall 2011)• Interagency reviewed & revised
Distribute Survey (Jan 30, 2012)• Distributed via SurveyMonkey to all QM staff at all provider agencies that participated in QM program in 2011
Analyze results (March-April 2012)• Quantitative—Descriptive statistics (%, ANOVA)• Qualitative – Open coding & Axial coding
Quantitative Results
Respondent Characteristics
Transitional Care Coordination (TCC)
Supportive counseling/family stabilization (SCF)
Outreach for homeless youth (OHY)
Mental Health (MSV)
Medical Case Management/Care Coordination- NYC (MCM)
Medical Case Management- Tri-County (MCM)
Harm Reduction Readiness & Relapse Prev (HRR)
Food and nutrition (FNS)
Early Intervention Services (EIS)
0% 5% 10% 15% 20% 25% 30%
1.2%
5.4%
1.8%
9.0%
28.1%
14.4%
20.4%
10.2%
9.6%
Percentage
Survey Respondents’ Characteristics
Community Health Center12.0%
Commu-nity
Based Orga-niza-tion
58.7%
Hos-pital
29.3%
Agency Type
• 7.93 years• Range = 0-25 years
Average Years
Worked
• 16.8% Executive Leaders• 67.1% Other• Program Director• Program Manager• Care Coordinator
Role at agency
Overall Satisfaction with QM Program*
1
1.5
2
2.5
3
3.5
4
4.5
5
3.80
4.13
3.623.95 3.76 3.86
3.33
3.66
4.50
Rat
ing
Aver
age
*No difference in means by service category, F = 0.98, p = 0.45
Overall 3.8
Satisfaction with Technical Assistance (TA)
1. No difference in means by service category, p > 0.052. TCC & OHY excluded because n<5
EIS
FNS
HRR
MCM NYC
MCM Tri-C
ounty MSV
SCF
1
1.5
2
2.5
3
3.5
4
4.5
5
3.73 3.67 3.64 3.683.89
4.10 4.00
TA helped create QM Plan 1,2
1
1.5
2
2.5
3
3.5
4
4.5
5
3.73 3.783.32
3.62 3.533.80
4.00
TA tailored to program needs 1,2
Overall3.58
Overall 3.75
Usefulness/Frequency of QM Activities
Very Useful
• LN Meetings • QI Trainings • Performance Review
Reports • Technical Assistance • Organizational
Assessments
Somewhat Useful
• Webinars/Teleconferences• Technical Assistance • Performance Review
Reports • Organizational
Assessments • LN Meetings
Not Useful
• Webinars/Teleconferences
• Organizational Assessments
• LN Meetings
Neither Increase/Decrease QM Activities• Communication (69.0%)• Webinars (66.7%)• QLN (64.3%)• Performance Monitoring (61.9%)• Coaching (54.8%)
Organizational Assessments (OA)
EIS HRR MCM NYC MCM Tri-County1
1.52
2.53
3.54
4.55
4.50
3.584.16 4.13
Feedback from OA helpful to program, N = 58
Overall3.94
1. No difference in means by service category, p > 0.052. TCC, FNS, MH, SCF removed because n<5
52.7%
12.7%
34.5%
Site receive OA, N = 110
YesNoNot sure
OA led to new strategies (N = 57)
HRR MCM NYC MCM Tri-County1
1.5
2
2.5
3
3.5
4
4.5
5
3.373.11
4.00
1. No difference in means by service category, p > 0.052. EIS, FNS, MH, SCF & TCC removed because n<5
Overall3.42
Usefulness of Performance Reports1,2
EIS MSV MCM-Tri-County FNS MCM-NYC HRR SCF1
1.5
2
2.5
3
3.5
4
4.5
5
3.71
3.33
3.91
3
3.84
3.12
4.25
N=60
Rat
ing
Aver
age
Overall3.58
1.No difference in means by service category, p > 0.052. TCC, OHY excluded n <5
Performance Reports, continued1,2
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
3.86
3.253.50
4.174.00 4.00
Relevancy of indicators
Overall3.85
EIS
FNS
HRR
MCM NYC
MCM Tri-C
ounty MSV
11.5
22.5
33.5
44.5
5
3.433.00
3.693.42 3.25
2.60
IPRO/NYCHSRO flexible
Overall3.42
1. No difference in means by service category, p > 0.052. SCF, TCC & OHY excluded because n<5
Performance Reports,
continued1,2
1
1.5
2
2.5
3
3.5
4
4.5
5
3.573.25
2.88
3.673.50
4.00
Reviews reflect performance
Overall3.38
1
1.5
2
2.5
3
3.5
4
4.5
5
3.14 3.00
2.44
3.583.50
3.00
Reports timely
Overall3.80
1. No difference in means by service category, p > 0.052. SCF, TCC & OHY excluded because n<5
Training topic Preferences• Quality improvement tools
1st Choice
• Communicating goals & Successes
• Planning & Implementing a QI project
• Using performance measurement
2nd
Choice
• Sustaining QI program3rd
Choice
4th Choice
• Gaining leadership support• Consumer involvement
Qualitative Results
Most Useful Webinars• n/a• None• Not useful
1st Choice30%
• Storyboard Development• Quality Learning Network
2nd
Choice8.5%
• Quality Management Planning
3rd Choice7.4%
4th Choice6.3%
• Quality Improvement Topics
Examples of New Approaches • PDSA• Storyboard
1st Choice11.5%
• Data Systems2nd
Choice7.6%
• Full agency CQI Team• Regular CQI Meetings
3rd Choice6.4%
4th Choice5.1%
• Retention Tool• QI as priority• New program forms
Quotes• We have made QM a priority and have a dedicated
steering committee that meets monthly.
• The program now has an eye to improving patient care and looking at specific goals to work on that we feel will improve patient care and health.
• I would say the main thing now is having the knowledge and skill to look at something, focus on it, and work towards improving it, and even if it does not generate the desired outcome, it is still a learning experience.
How LNL helped accomplish QI goals• Tools• Resources• Guidance• Framework
1st Choice24.6%
• TA on-site and on phone
2nd
Choice10.0%
• Not yet, just started• Support and feedback
3rd Choice8.7%
4th Choice7.2%
• Learn from other agencies
Quotes•The LNL provided guidance and
resources to develop and implement quality improvement strategies that were not available to members or that we did not have knowledge of. These tools were extremely helpful in improving our skills in quality improvement and shift our paradigm about determining quality measures and what to evaluate in our programs.
Quotes•The QLN inspired our QI team to utilize
fishbone diagrams when working on our QI project.
•The LNL came to our site and gave us feedback on how were doing overall.
Quotes•It helped us to think outside the box and
to separate what PHS expects from the program in terms of numbers with an overall and more general view on how services are provided and what is not working. We were then able to make appropriate changes to different areas of the program to make it more successful.
Additional Comments• n/a• None
1st Choice20%
• Helpful• Useful
2nd
Choice13.3%
• Good Networking3rd
Choice6.7%
Quotes•The learning network is a very helpful
networking program. I look forward to going there and know that I can get assistance with challenges that I may be experiencing in the program.
•It is a great opportunity to learn about what other providers are doing and also for us to share what has worked in our program.
How program used performance reports
• Helped us focus on one area
1st Choice20.5%
• Helped us improve our forms
• Led to QM strategies2nd Choice
11.4%
Quotes•Upon receiving the quality reports we
were able to focus on areas which needed improvement. One example was intakes/assessments. We were able to communicate our needs to other providers in which referrals were being funneled by several agencies and helped us to meet our goal.
Quotes•We were able to look specifically at our
screenings for STI and substance use to review for any possible edits and look at best practices when aligned with other agencies.
•Used reports to identify any program shortcomings and then developed a QM project to address the findings.
Ideas for Improvement - Themes
Meeting logistics
People should attend
regularly
Repetitive information
Setting annual goals for Learning
Network
Overlap of meetings
and groupsPerformance Reports
Meeting Logistics
Meeting Logistics
Time
Snacks
Location
Frequency
Meeting Logistics: Time•Meetings smack in the middle of the day
are very inconvenient
•It would be helpful if meetings could be scheduled to either start at the beginning of the day (9:00am or 9:30am) or end at the end of the day (4pm or 5pm)
•Meetings should be for a half-day only
Meeting Logistics, Cont.Frequency
• I believe that trainings should be offered twice a year rather than four or more times
Snacks
• Breakfast should also include (if any) cold drinks for those who do not drink or cannot drink coffee
Location
• Meetings should be done in Upper NYC or Bronx at times.
Regular Attendance
• The meetings might be more useful if agencies send the same people and each gets to present at least once yearly
Repetitive Information
• At times webinars and network meetings have felt somewhat redundant.
Setting annual goals for Learning Network
Also it would be helpful if there were annual goals for learning networks so that expectations for what will be accomplished for the year are clear to everyone participating (if sub-grantees are expected to have plans, so should the learning network)
Overlap of meetings and groupsWhen referring to the various quality improvement projects, it is extremely difficult figuring out what group of people you are referring to. For example, the HRR LN would be one group of people. Do they fall under the AI? Another group of people (NYSCHRO) usually comes in to collect data, are we referring to them? The UMRG sometimes facilitated by Susan Weigl, are you referring to the QI projects at this meeting? Too much overlap which makes it confusing to respond appropriately.
Performance Indicators and Reports• Select far fewer indicators and make sure they are
clearly defined. Right now, there are too many indicators, and it’s not at all clear that all programs define them in the same way. And whatever indicators are selected should be linked to program outcomes.
• NYCSHRO reviews require way too much time and attention and I don’t think the resources required to conduct them pay off in any way that’s beneficial to the portfolio.
• Performance reports given to us in a more timely fashion
Limitations•Low response rate (24.6%)
▫Response bias▫Generalizability
•People responding to questions having no experience with QLNs (OHY & TCC)▫We only included the Part A providers and
several Statewide providers are part of the LN--you should probably note that
Lessons Learned & Next Steps•Appropriate of QM activities•Review IPRO/NYCHSRO scheduling
procedures•Review reporting deadlines•Implement simple solutions
▫Meeting logistics
Questions? Thoughts?
Acknowledgements•Respondents•Interagency group
NYSDOH AIDS Institute: Tracy Hatton, MPH; Nova West, MPH; Susan Weigl, MPH;Johanna Buck, RN; Bruce Agins, MD, MPH
Public Health Solutions: Bettina CaroleRachel Miller, MPH
NYC DOHMH: Graham Harriman, MA ;Beau Mitts, MPH
Westchester Dept of Health: Tom Petro Julie Lehane, PhD
Obtaining CME/CE Credit•If you would like to receive continuing
education credit for this activity, please visit:
•http://www.pesgce.com/RyanWhite2012
THANK YOU!
Heather Mavronicolas, PhD, MPHDirector, Quality Management & Special ProjectsBureau of HIV/AIDS Prevention and [email protected]
Mary Kay Diakite, LMSWTechnical Assistance Project ManagerBureau of HIV/AIDS Prevention and [email protected]