Download - Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods
![Page 1: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/1.jpg)
Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods
Ryan White All Grantee MeetingNovember 29,2012
Presenters: Mary Kay Diakite, LMSW and Graham Harriman, MAHIV Care, Treatment and Housing ProgramNYC Department of Health & Mental Hygiene
![Page 2: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/2.jpg)
Disclosures• This continuing education activity is managed
and accredited by Professional Education Service Group. The information presented in this activity represents the opinion of the author(s) or faculty. Neither PESG, nor any accrediting organization endorses any commercial products displayed or mentioned in conjunction with this activity.
• Commercial Support was NOT received for this activity.
![Page 3: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/3.jpg)
Disclosures•Mary Kay Diakite, LMSW
• Has no financial interest or relationships to disclose
•Graham Harriman, MA• Has no financial interest or relationships to
disclose
![Page 4: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/4.jpg)
Learning ObjectivesAt the end of this activity, the participant will be able to:
1. Apply quantitative and qualitative methods in quality management program evaluations
2. Determine the most appropriate methods to apply in their particular EMA
![Page 5: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/5.jpg)
Agenda•Background•Objectives•Methodology•Results
▫Quantitative▫Qualitative
•Discussion
![Page 6: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/6.jpg)
Background• Quality management (QM) program goals
▫ Comprehensive performance measurement systems are developed & utilized to assess the quality of Part A services
▫ Enable Part A providers to continue to be skilled in using QI tools & methodologies to enhance the quality of their services
▫ Part A services improve health outcomes and quality of life for people living with HIV/AIDS
▫ The Part A QM Program remains responsive to the changing science & epidemiologic needs of the NY EMA community
![Page 7: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/7.jpg)
Quality Management Activities
Organizational
Assessments
Performance Measuremen
t
Quality Management
Plans
Quality Learning Networks
![Page 8: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/8.jpg)
Evaluation Objectives
Assess the Ryan White Part A QM Program conducted through the NYSDOH AIDS Institute during Jan-Dec 2011
Identify specific areas for improvement
Identify best practices & plan future trainings
![Page 9: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/9.jpg)
Survey Overview
General perception
sAgenc
y
OALN
meetings
QI training
s
Performance
Indicators
![Page 10: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/10.jpg)
Methodology & Timeline
Survey Development (Fall 2011)• Interagency reviewed & revised
Distribute Survey (Jan 30, 2012)• Distributed via SurveyMonkey to all QM staff at all provider agencies that participated in QM program in 2011
Analyze results (March-April 2012)• Quantitative—Descriptive statistics (%, ANOVA)• Qualitative – Open coding & Axial coding
![Page 11: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/11.jpg)
Quantitative Results
![Page 12: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/12.jpg)
Respondent Characteristics
Transitional Care Coordination (TCC)
Supportive counseling/family stabilization (SCF)
Outreach for homeless youth (OHY)
Mental Health (MSV)
Medical Case Management/Care Coordination- NYC (MCM)
Medical Case Management- Tri-County (MCM)
Harm Reduction Readiness & Relapse Prev (HRR)
Food and nutrition (FNS)
Early Intervention Services (EIS)
0% 5% 10% 15% 20% 25% 30%
1.2%
5.4%
1.8%
9.0%
28.1%
14.4%
20.4%
10.2%
9.6%
Percentage
![Page 13: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/13.jpg)
Survey Respondents’ Characteristics
Community Health Center12.0%
Commu-nity
Based Orga-niza-tion
58.7%
Hos-pital
29.3%
Agency Type
• 7.93 years• Range = 0-25 years
Average Years
Worked
• 16.8% Executive Leaders• 67.1% Other• Program Director• Program Manager• Care Coordinator
Role at agency
![Page 14: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/14.jpg)
Overall Satisfaction with QM Program*
1
1.5
2
2.5
3
3.5
4
4.5
5
3.80
4.13
3.623.95 3.76 3.86
3.33
3.66
4.50
Rat
ing
Aver
age
*No difference in means by service category, F = 0.98, p = 0.45
Overall 3.8
![Page 15: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/15.jpg)
Satisfaction with Technical Assistance (TA)
1. No difference in means by service category, p > 0.052. TCC & OHY excluded because n<5
EIS
FNS
HRR
MCM NYC
MCM Tri-C
ounty MSV
SCF
1
1.5
2
2.5
3
3.5
4
4.5
5
3.73 3.67 3.64 3.683.89
4.10 4.00
TA helped create QM Plan 1,2
1
1.5
2
2.5
3
3.5
4
4.5
5
3.73 3.783.32
3.62 3.533.80
4.00
TA tailored to program needs 1,2
Overall3.58
Overall 3.75
![Page 16: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/16.jpg)
Usefulness/Frequency of QM Activities
Very Useful
• LN Meetings • QI Trainings • Performance Review
Reports • Technical Assistance • Organizational
Assessments
Somewhat Useful
• Webinars/Teleconferences• Technical Assistance • Performance Review
Reports • Organizational
Assessments • LN Meetings
Not Useful
• Webinars/Teleconferences
• Organizational Assessments
• LN Meetings
Neither Increase/Decrease QM Activities• Communication (69.0%)• Webinars (66.7%)• QLN (64.3%)• Performance Monitoring (61.9%)• Coaching (54.8%)
![Page 17: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/17.jpg)
Organizational Assessments (OA)
EIS HRR MCM NYC MCM Tri-County1
1.52
2.53
3.54
4.55
4.50
3.584.16 4.13
Feedback from OA helpful to program, N = 58
Overall3.94
1. No difference in means by service category, p > 0.052. TCC, FNS, MH, SCF removed because n<5
52.7%
12.7%
34.5%
Site receive OA, N = 110
YesNoNot sure
![Page 18: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/18.jpg)
OA led to new strategies (N = 57)
HRR MCM NYC MCM Tri-County1
1.5
2
2.5
3
3.5
4
4.5
5
3.373.11
4.00
1. No difference in means by service category, p > 0.052. EIS, FNS, MH, SCF & TCC removed because n<5
Overall3.42
![Page 19: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/19.jpg)
Usefulness of Performance Reports1,2
EIS MSV MCM-Tri-County FNS MCM-NYC HRR SCF1
1.5
2
2.5
3
3.5
4
4.5
5
3.71
3.33
3.91
3
3.84
3.12
4.25
N=60
Rat
ing
Aver
age
Overall3.58
1.No difference in means by service category, p > 0.052. TCC, OHY excluded n <5
![Page 20: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/20.jpg)
Performance Reports, continued1,2
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
3.86
3.253.50
4.174.00 4.00
Relevancy of indicators
Overall3.85
EIS
FNS
HRR
MCM NYC
MCM Tri-C
ounty MSV
11.5
22.5
33.5
44.5
5
3.433.00
3.693.42 3.25
2.60
IPRO/NYCHSRO flexible
Overall3.42
1. No difference in means by service category, p > 0.052. SCF, TCC & OHY excluded because n<5
![Page 21: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/21.jpg)
Performance Reports,
continued1,2
1
1.5
2
2.5
3
3.5
4
4.5
5
3.573.25
2.88
3.673.50
4.00
Reviews reflect performance
Overall3.38
1
1.5
2
2.5
3
3.5
4
4.5
5
3.14 3.00
2.44
3.583.50
3.00
Reports timely
Overall3.80
1. No difference in means by service category, p > 0.052. SCF, TCC & OHY excluded because n<5
![Page 22: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/22.jpg)
Training topic Preferences• Quality improvement tools
1st Choice
• Communicating goals & Successes
• Planning & Implementing a QI project
• Using performance measurement
2nd
Choice
• Sustaining QI program3rd
Choice
4th Choice
• Gaining leadership support• Consumer involvement
![Page 23: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/23.jpg)
Qualitative Results
![Page 24: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/24.jpg)
Most Useful Webinars• n/a• None• Not useful
1st Choice30%
• Storyboard Development• Quality Learning Network
2nd
Choice8.5%
• Quality Management Planning
3rd Choice7.4%
4th Choice6.3%
• Quality Improvement Topics
![Page 25: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/25.jpg)
Examples of New Approaches • PDSA• Storyboard
1st Choice11.5%
• Data Systems2nd
Choice7.6%
• Full agency CQI Team• Regular CQI Meetings
3rd Choice6.4%
4th Choice5.1%
• Retention Tool• QI as priority• New program forms
![Page 26: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/26.jpg)
Quotes• We have made QM a priority and have a dedicated
steering committee that meets monthly.
• The program now has an eye to improving patient care and looking at specific goals to work on that we feel will improve patient care and health.
• I would say the main thing now is having the knowledge and skill to look at something, focus on it, and work towards improving it, and even if it does not generate the desired outcome, it is still a learning experience.
![Page 27: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/27.jpg)
How LNL helped accomplish QI goals• Tools• Resources• Guidance• Framework
1st Choice24.6%
• TA on-site and on phone
2nd
Choice10.0%
• Not yet, just started• Support and feedback
3rd Choice8.7%
4th Choice7.2%
• Learn from other agencies
![Page 28: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/28.jpg)
Quotes•The LNL provided guidance and
resources to develop and implement quality improvement strategies that were not available to members or that we did not have knowledge of. These tools were extremely helpful in improving our skills in quality improvement and shift our paradigm about determining quality measures and what to evaluate in our programs.
![Page 29: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/29.jpg)
Quotes•The QLN inspired our QI team to utilize
fishbone diagrams when working on our QI project.
•The LNL came to our site and gave us feedback on how were doing overall.
![Page 30: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/30.jpg)
Quotes•It helped us to think outside the box and
to separate what PHS expects from the program in terms of numbers with an overall and more general view on how services are provided and what is not working. We were then able to make appropriate changes to different areas of the program to make it more successful.
![Page 31: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/31.jpg)
Additional Comments• n/a• None
1st Choice20%
• Helpful• Useful
2nd
Choice13.3%
• Good Networking3rd
Choice6.7%
![Page 32: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/32.jpg)
Quotes•The learning network is a very helpful
networking program. I look forward to going there and know that I can get assistance with challenges that I may be experiencing in the program.
•It is a great opportunity to learn about what other providers are doing and also for us to share what has worked in our program.
![Page 33: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/33.jpg)
How program used performance reports
• Helped us focus on one area
1st Choice20.5%
• Helped us improve our forms
• Led to QM strategies2nd Choice
11.4%
![Page 34: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/34.jpg)
Quotes•Upon receiving the quality reports we
were able to focus on areas which needed improvement. One example was intakes/assessments. We were able to communicate our needs to other providers in which referrals were being funneled by several agencies and helped us to meet our goal.
![Page 35: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/35.jpg)
Quotes•We were able to look specifically at our
screenings for STI and substance use to review for any possible edits and look at best practices when aligned with other agencies.
•Used reports to identify any program shortcomings and then developed a QM project to address the findings.
![Page 36: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/36.jpg)
Ideas for Improvement - Themes
Meeting logistics
People should attend
regularly
Repetitive information
Setting annual goals for Learning
Network
Overlap of meetings
and groupsPerformance Reports
![Page 37: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/37.jpg)
Meeting Logistics
Meeting Logistics
Time
Snacks
Location
Frequency
![Page 38: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/38.jpg)
Meeting Logistics: Time•Meetings smack in the middle of the day
are very inconvenient
•It would be helpful if meetings could be scheduled to either start at the beginning of the day (9:00am or 9:30am) or end at the end of the day (4pm or 5pm)
•Meetings should be for a half-day only
![Page 39: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/39.jpg)
Meeting Logistics, Cont.Frequency
• I believe that trainings should be offered twice a year rather than four or more times
Snacks
• Breakfast should also include (if any) cold drinks for those who do not drink or cannot drink coffee
Location
• Meetings should be done in Upper NYC or Bronx at times.
![Page 40: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/40.jpg)
Regular Attendance
• The meetings might be more useful if agencies send the same people and each gets to present at least once yearly
Repetitive Information
• At times webinars and network meetings have felt somewhat redundant.
![Page 41: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/41.jpg)
Setting annual goals for Learning Network
Also it would be helpful if there were annual goals for learning networks so that expectations for what will be accomplished for the year are clear to everyone participating (if sub-grantees are expected to have plans, so should the learning network)
![Page 42: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/42.jpg)
Overlap of meetings and groupsWhen referring to the various quality improvement projects, it is extremely difficult figuring out what group of people you are referring to. For example, the HRR LN would be one group of people. Do they fall under the AI? Another group of people (NYSCHRO) usually comes in to collect data, are we referring to them? The UMRG sometimes facilitated by Susan Weigl, are you referring to the QI projects at this meeting? Too much overlap which makes it confusing to respond appropriately.
![Page 43: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/43.jpg)
Performance Indicators and Reports• Select far fewer indicators and make sure they are
clearly defined. Right now, there are too many indicators, and it’s not at all clear that all programs define them in the same way. And whatever indicators are selected should be linked to program outcomes.
• NYCSHRO reviews require way too much time and attention and I don’t think the resources required to conduct them pay off in any way that’s beneficial to the portfolio.
• Performance reports given to us in a more timely fashion
![Page 44: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/44.jpg)
Limitations•Low response rate (24.6%)
▫Response bias▫Generalizability
•People responding to questions having no experience with QLNs (OHY & TCC)▫We only included the Part A providers and
several Statewide providers are part of the LN--you should probably note that
![Page 45: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/45.jpg)
Lessons Learned & Next Steps•Appropriate of QM activities•Review IPRO/NYCHSRO scheduling
procedures•Review reporting deadlines•Implement simple solutions
▫Meeting logistics
![Page 46: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/46.jpg)
Questions? Thoughts?
![Page 47: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/47.jpg)
Acknowledgements•Respondents•Interagency group
NYSDOH AIDS Institute: Tracy Hatton, MPH; Nova West, MPH; Susan Weigl, MPH;Johanna Buck, RN; Bruce Agins, MD, MPH
Public Health Solutions: Bettina CaroleRachel Miller, MPH
NYC DOHMH: Graham Harriman, MA ;Beau Mitts, MPH
Westchester Dept of Health: Tom Petro Julie Lehane, PhD
![Page 48: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/48.jpg)
Obtaining CME/CE Credit•If you would like to receive continuing
education credit for this activity, please visit:
•http://www.pesgce.com/RyanWhite2012
![Page 49: Evaluating a Ryan White Quality Management Program using quantitative and qualitative methods](https://reader035.vdocuments.mx/reader035/viewer/2022062323/56816567550346895dd7f1d2/html5/thumbnails/49.jpg)
THANK YOU!
Heather Mavronicolas, PhD, MPHDirector, Quality Management & Special ProjectsBureau of HIV/AIDS Prevention and [email protected]
Mary Kay Diakite, LMSWTechnical Assistance Project ManagerBureau of HIV/AIDS Prevention and [email protected]