assessment and progress monitoring in school-based mental health
DESCRIPTION
Assessment and Progress Monitoring in School-Based Mental Health. A collaboration between Public Health of Seattle-King Country, the Seattle Public Schools, and the University of Washington School of Medicine/Seattle Children’s Hospital. Aaron Lyon, PhD (UW/SCH) - PowerPoint PPT PresentationTRANSCRIPT
Assessment and Progress Monitoring in School-Based
Mental Health
October 16, 2012
Aaron Lyon, PhD (UW/SCH)Jessica Knaster Wasse, MPH (PHSKC)
Kristy Ludwig, PhD (UW/SCH)
A collaboration between Public Health of Seattle-King Country, the Seattle Public Schools, and the University of Washington School of Medicine/Seattle
Children’s Hospital
Acknowledgments
• Ann Vander Stoep• Elizabeth McCauley• Eric Bruns• Evalynn Romano• Jane Koltracht• Kelly Thompson• Seattle Public Schools• TJ Cosgrove• School-based practitioners!
Collaborators: Funding:• National Institute of Mental
Health (K08MH095939)• Institute of Education Sciences
(R305A120128)• American Psychological
Foundation• Public Health of Seattle & King
County• Bill and Melinda Gates
Foundation
Dr. Lyon is an investigator with the Implementation Research Institute (IRI), at the George Warren Brown School of Social Work, Washington University in St. Louis; through an award from the National Institute of Mental Health (R25MH080916-01A2) and the Department of Veterans Affairs, Health Services Research & Development Service, Quality Enhancement Research Initiative (QUERI).
Overview1. Background & goals of standardized
assessment / progress monitoring initiative in Seattle’s school-based health centers (SBHCs)
2. Public Health of Seattle & King County’s (PHSKC) interest in / support for progress monitoring in SBHCs
3. Current training approach / activities4. Future directions
School-Based Health Centers (SBHCs)
• Integrated care clinics situated in schools• Operate in nearly 2,000 schools in the US (NASBH,
2008)
• Typically provide primary care and mental health services (Brown & Bolen, 2003)
• Well-substantiated as a mechanism to increase service accessibility to underserved and under/uninsured (Gance-Cleveland & Yousey, 2005; Kaplan et al., 1999; Wade et al., 2008)
Evidence-Based Practice in SBMH
• EBP utilized inconsistently in SBMH (Evans & Weist, 2004)
• Implementation of EBP in SBMH should focus on low-cost, high-yield practice improvement targets• Disseminate / Implement key competencies instead of full Tx
packages (Beidas et al., 2011; Rotheram-borus et al., 2012)
• Evidence-based assessment may be the “low-hanging fruit” of EBP implementation• School-based providers unlikely to use standardized assessment
and progress monitoring routinely• Previous work has demonstrated good uptake of SA following
training (Lyon et al., 2011)
Evidence-Based Assessment
Evidence-Based Assessment (EBA): “Assessment methods and processes that are based on empirical evidence in terms of both their reliability and validity as well as their clinical usefulness for prescribed populations and purposes” (Mash & Hunsley, 2005, p.364)• Evidence-based interventions commonly rely on EBA
Standardized assessment (SA), or the use of measurement tools with empirical support for their reliability, validity, etc, is at the core of EBA (Jensen-Doss & Hawley, 2005)
Value of Standardized AssessmentInitial Assessment• Rating scales can increase the ease and accuracy of
clinical diagnosis (e.g., Jenkins et al., 2011; Youngstrom et al., 2005)
• Psychological assessment carries positive, clinically meaningful effects (Posten & Hanson, 2010)
Progress Monitoring• Clinicians are often not able to detect client
deterioration (Hannan et al., 2005)
• Providing assessment results to clinicians can result in improved outcomes (Bickman et al., 2011; Lambert et al., 2003)
Steps of Standardized Assessment Use
1. Selection2. Administration3. Scoring4. Interpretation5. Feedback to clients6. Treatment planning7. Monitoring (over time w/ feedback)8. Treatment plan adjustment
SA / Progress Monitoring: SBMH v. CMHSBHC
(n = 16)CMH
(n = 58)
Percent of Caseload: 0% 1-39% 0% 1-39%
Gave a SA measure in initial 1-2 meetings 47% 27% 16% 32%
Gave a SA measure at termination 63% 36% 52% 23%
Gave a SA measure 38% 44% 42% 42%
Gave feedback about a SA measure 50% 31% 46% 30%
Changed Tx plan based on SA data 69% 31% 64% 32%
Changed indiv. session plan based on SA 56% 38% 64% 27%
Progress Monitoring & Feedback• Progress monitoring may use SA tools or ideographic
monitoring targets• Value of ideographic targets (Weisz et al., 2011)…
• Add specificity to ID’d problems• Focus on client concerns / priorities• Give clients a voice• Enhance rapport / alliance• Provide foci for ongoing assessment• Provide info about treatment termination• Easy to fit into everyday practice
• Combination of SA and ideographic is likely optimal
Progress Monitoring & Feedback• Feedback Intervention Theory (Kluget & DeNisi, 1996): Bx is
regulated by comparisons of feedback to goals• Two levels (at least!): (1) feedback to the clinician & (2)
feedback to the client• Most research has focused on feedback to clinicians
• Feedback is associated with higher rapport with provider, client self-understanding (Allen et al., 2003), and increased engagement in therapy (Ackerman et al., 2000)
• Academic outcomes are an essential domain for youth success, but rarely included in MH research (Atkins et al., 2010; Franklin et al., 2009; Hoagwood et al., 2007)
• Youth w/ combined MH and academic risk have a greater likelihood of long-term negative outcomes (Roeser et al., 1999)
• School-based clinicians rate school attendance as the most clinically-useful academic indicator (Lyon, Ludwig, & McCauly, in prep)
Monitoring School Outcomes in Tx
Potential Assessment / Monitoring Targets
Mental Health•Depression symptoms•Disruptive behavior•Self-injurious behavior
School Engagement•Attendance•Homework completion•Class participation•School connectedness
Academic Outcomes•Grades•Credits earned•Standardized test scores
Social Functioning•Interpersonal conflicts•Positive social experiences•Disciplinary events
Physical Health•Sleep•Diet & Exercise
Services•Satisfaction with treatment•Engagement in intervention
1. Engage stakeholders in planning / adapting a computerized measurement feedback system (MFS) to support SBMH providers
2. Provide training to SBMH providers in the use of standardized and ideographic assessment / progress monitoring
3. Implement, evaluate, and iterate the MFS, incorporating the ability to monitor academic outcomes
Current Goals & Activities
Engaging Stakeholders in the Adaptation of a Measurement Feedback System for SBHCs
Policy/funding context •Seattle Families & Education Levy
• Property tax levy, renewed for 4th 7 year term starting 2012• SBHC funding managed by PHSKC• Levy performance goals are academic in nature
•Successfully lobbied for Levy $ for MH enhancement• Goal: Enhance the academic impact of Levy health investments
by improving the quality of school mental health services • Increase use of evidence-based practice, with focus on
standardized assessment and outcome monitoring
SBHCs in Seattle / King County
SBHCs in Seattle / King County
Vision: Collaborative stepped care model• SA data used to support clinical decision making• Track clinical improvement and identify patients who are not
improving as expected
Aligns with Seattle Public Schools’ Response to Intervention (RtI) strategy
Graphic from the National Center on Response to Intervention
• “When developing an evaluation system, two of the most challenging components are getting buy-in from all partners involved and then selecting the measurement tools” (Sander et al., 2011, p.15).
• Gates Foundation funding for one-year planning process• Stakeholder committee comprised of: school-based MH
providers, supervisors, evaluators, program managers, computer programmers, school district
Stakeholder Committee
Monthly meetings designed to:
• Build initial consensus about data fields and other user-interface issues
• Identify initial battery of standardized assessment measures
• Identify SA toolkit for various MH domains• Develop and refine implementation / evaluation plan• Enhance provider buy-in
Stakeholder Committee
Challenges/hurdles
• Differing documentation requirements• Duplicate data entry EHR and MFS• Varying degrees of technical knowledge• Evaluation design: impact of randomization
Stakeholder Committee
Results/Successes
•Data sharing agreement with Seattle Public Schools•Specifications completed on schedule•Negotiated acceptable evaluation plan•Providers engaged and excited
Stakeholder Committee
Training in Standardized Assessment and Progress
Monitoring
In promoting data-based decision making, SBMH professionals often struggle to (Kelly, 2011)…
1. Clearly define the reason for collecting the data2. Ensure that data-collection procedures are manageable / user-friendly3. Map out a clear system to use the data for continuous improvement
Training in SA & Progress Monitoring
• Over the past 2 years, our group has provided a training and consultation series to local SBHC MH providers covering…• Standardized assessment principles, psychometrics,
and normative data• Initial assessment using standardized tools• Feedback to students on the basis of assessment• Identifying and tracking progress monitoring targets
Training in SA & Progress Monitoring
Principles of progress monitoring:•Monitoring of some type is a good idea for ALL youth receiving services•Only track those targets that are meaningful to the specific case/match treatment goals•Graphical feedback increases
understanding and makes the feedback more memorable (Kluger & DiNisi, 1996; Miller & Watkins, 2010)
•Can be used as a tool to validate changes in a student’s experience
Training in SA & Progress Monitoring
Training in SA & Progress Monitoring
Moving from the WHAT to WHY… •Progress monitoring is the first step and gives you the WHAT•Next steps include identifying the WHY (i.e., why is a target changing or not changing?)
• Moving beyond just “success” or “failure”
•The WHY is important to the kid feeling successful• Kid’s feelings of agency and control (therapy isn’t being done to
them or for them but something they are willfully and actively participating in)
Assessment Flow Chart
1) Decrease anxiety symptoms
2) Improved school engagement
3) Improved sleep hygiene
Student Presentation Assess
Treatment Goal ID
SA
Ideo-graphic
SA: MASC-10 IG: Anxiety experienced when arriving at school?
SA: School Connectedness ScaleIG: # on-time arrivals for 1st periodSA: None?IG: Nights/wk w/o FB or texting after 8pm
Current Findings and Future Evaluation Directions
Focus Groups • Focus groups conducted to evaluate the context in which the MFS
will be implemented.• Transcribed and evaluated using a conventional content analysis
approach. • EMERGING THEME: Factors which facilitate or inhibit the utility
of a technology in SBMH (Lyon et al., in prep)• Accessibility• Impact on Youth Engagement• Confidentiality• Knowledge or Attitudes (Provider)• Integration of Aspects of Care• Relevance to MH Care• Speed of the Technology
Pilot of the Un-adapted MFS
• Anonymous surveys of youth receiving SBMH services supported by the un-adapted MFS (n = 34)• 71% of youth respondents stated that answering questionnaires
about emotions made it easier to tell their providers how they were feeling
• 55% said answering the questionnaires (and receiving feedback) made the time spent with their SBMH providers either a little or much easier/better.
• 45% said it did not affect their interactions (no students reported that it made it worse)
Many Questions Remain…Selected findings from Key Informant individual interviews from a related project…•Who should provide feedback about progress?
• Teachers, students, school teams
•Who should receive / review feedback?• In addition to clinician, principal, teachers, school staff, the
academic counselor, school teams• Allow student to determine who will review
•Other challenges• Unique data systems and data sharing rules
• May require individualized approach district-by-district• Providers not used to collecting/using data systematically• Transitioning from health/MH to academic outcomes
Next Steps
1. Complete initial adaption of the MFS2. Implement the MFS with randomly-selected providers3. Compare provider behavior (e.g., use of standardized or
ideographic assessment/monitoring) between MFS and non-MFS groups
4. Conduct independent evaluations of youth outcomes by recruiting from the caseloads of MFS and non-MFS providers
Questions and Discussion