wraparound fidelity monitoring in the massachusetts cbhi summary of 2011 results august 12, 2011

Wraparound Fidelity Wraparound Fidelity Monitoring in the Monitoring in the

Massachusetts CBHIMassachusetts CBHI

Summary of 2011 ResultsSummary of 2011 Results

August 12, 2011

Eric Bruns and April SatherUniversity of WashingtonWraparound Evaluation and Research [email protected] / [email protected] thanks to Andrea Gewirtz and Melissa King

Purpose for this MeetingPurpose for this Meeting

1.1. Very quick review of wraparound Very quick review of wraparound fidelity assessment and researchfidelity assessment and research

2.2. Review data from Massachusetts Review data from Massachusetts fidelity monitoringfidelity monitoring

3.3. Discuss what has been found and what Discuss what has been found and what it meansit means

Wraparound Principles

1. Family voice and choice2. Team-based3. Natural supports4. Collaboration5. Community-based6. Culturally competent7. Individualized8. Strengths based9. Persistence10.Outcome-based

Walker, Bruns, Adams, Miles, Osher et al., 2004

Wraparound Competence:What do we want to measure?

Implementing the practice model:

The Four Phases of Wraparound

Time

Engagement and Support

Team Preparation

Initial Plan Development

Implementation

Transition

Phase1A

Phase1B

Phase2

Phase3

Phase4

Wraparound Competence

Phase 1 : Engagement and Team Preparation

• Care Coordinator & Family Partner meets with the family to discuss the wraparound process and listen to the family’s story.

• Discuss concerns, needs, hopes, dreams, and strengths.

• Listen to the family’s vision for the future.

• Assess for safety and make a provisional safety plan if needed

• Identify people who care about the family as well as people the family have found helpful for each family member.

• Reach agreement about who will come to a meeting to develop a plan and where we should have that meeting.

Phase 1 A and BPhase 1 A and B

Phase 2: Initial Plan Development

• Conduct first Care Planning Team (CPT) meeting with people who are providing services to the family as well as people who are connected to the family in a supportive role.

• The team will:

– Review the family vision

– Develop a Mission Statement about what the team will be working on together

– Review the family’s needs

– Come up with several different ways to meet those needs that match up with the family’s strengths

• Different team members will take on different tasks that have been agreed to.

Phase 2Phase 2

Phase 3: Plan Implementation

• Based on the CPT meetings, the team has created a written plan of care.

• Action steps have been created, team members are committed to do the work, and our team comes together regularly.

• When the team meets, it:

– Reviews Accomplishments (what has been done and what’s been going well);

– Assesses whether the plan has been working to achieve the family’s goals;

– Adjusts things that aren’t working within the plan;

– Assigns new tasks to team members.

Phase 3Phase 3

Phase 4: Transition

• There is a point when the team will no longer need to meet regularly.

• Transition out of Wraparound may involve a final meeting of the whole team, a small celebration, or simply the family deciding they are ready to move on.

• The family we will get a record of what work was completed as well as list of what was accomplished.

• The team will also make a plan for the future, including who the family can call on if they need help or if they need to re-convene their team.

• Sometimes transition steps include the family and their supports practicing responses to crises or problems that may arise

Phase 4Phase 4

Wraparound Fidelity Index, v.4

• Items on the principles and core activities, organized by the 4 phases of wraparound

– Engagement: Did you select the people who would be on your youth and family team?

• Principle = Team based

– Planning: Does the plan include strategies for helping your child get involved with activities in the community?

• Principle = Community based

– Implementation: Does the team evaluate progress toward the goals of the plan at every team meeting?

• Principle = Outcome based

– Transition: Will some members of your team be there to support you when formal wraparound is complete?

• Principle = Persistence

Team Observation Measure

• Consists of 20 items, with two items dedicated to each of the 10 principles of wraparound.

• Each item consists of 3-5 indicators of high-quality wraparound practice as expressed during a care planning team meeting.

• Internal consistency very good

• Inter-rater reliability found to be adequate (Average 79% agreement for all indicators)

• Correlates well with WFI scores

– In previous studies and evaluations…

Document Review Measure

• Mass version consists of 26 items

• Each wraparound principle linked to several items

• Scale = 0-3, with criteria for each point on the scale

• Source material = documentation (electronic or paper) related to youth’s wraparound process

– Strengths, needs, culture discovery documentation

– Wraparound plan of care

– Safety plan

– Transition plan

– Progress notes

Measures of the WFAS seem to tap into the same constructs

50%

60%

70%

80%

90%

100%

WFI Combined 84% 77% 79% 79% 69% 72% 72% 81%

Team Observation 93% 84% 83% 78% 67% 56% 63% 78%

Site 1 Site 3 Site 5 Site 6 Site 7 Site 8 Site 9 Site 10

Mean WFI-4 and TOM total scores by site in one state’s QI project

WFAS Measures seem to be related to what we would expect

• Families who experience better outcomes have staff who score higher on fidelity tools (Bruns, Rast et al., 2006; Effland, McIntyre, & Walton, 2010)

• Wraparound initiatives with positive fidelity assessments demonstrate more positive outcomes (Bruns, Leverentz-Brady, & Suter, 2008)

• Wraparound initiatives with better support to implementation have higher fidelity scores

What does it take to get high fidelity scores?

• Training and coaching: Found to be associated with gains in fidelity and higher fidelity

• Community, Fiscal, and Service system support: Better developed supports = fidelity scores

• Sticking with it: Fidelity scores tend to increase over time and with persistence of effort

– But… they can also backslide…

Caregiver WFI Fidelity over time in NV

72%86%

73%64%

0%10%20%30%40%50%60%70%80%90%

100%

Pre Training(2001)

Training Only(2002)

Training andCoaching (2004)

After going toscale (2007-09)

Ave

rag

e W

FI

Fid

elit

y S

core

Bruns, Rast, Walker, Peterson, & Bosworth (2006). American Journal of Community Psychology.

What can we do with the data?

• Measurement as communication

– What are we attempting to do for families?

– How well are we stacking up against benchmarks? What is holding us back?

– How well are we doing over time?

– What is associated with positive outcomes in Massachusetts?

Measurement for quality improvement

• Examine data at the program level to discuss what is and is not happening consistently

• What can be reinforced in individual and group supervision?

• What are the team, program, and resource barriers?

• In what areas do we need additional training?

– At the community or system level by collaborative groups, along with reports of team, program, and resource barriers reported up from CSAs or programs

What kind of data get used?

• What kind of data?

– Item scores – Ns who endorse each response option (Yes, No, Somewhat)

– Members on teams

– Specific plan of care elements

– Qualitative feedback on open-ended items

Part 2: Results from Part 2: Results from MassachusettsMassachusetts

Scores on the WFI, TOM, and Mass-DRM

Total scores on the WFAS instruments

0

20

40

60

80

100

Percent of total fidelity

Massachusetts 2010 78 83 61

Massachusetts 2011 77 85 74

National mean 74 77 -

WFI TOM DRM

Correlations between WFI, TOM, and DRM across Massachusetts CSAs

• WFI – TOM = .121

• WFI – DRM = -.473 (p<.001)

• DRM – TOM = .012

• WFI – TOM = -.256

• WFI – DRM = .326 (p<.05)

• DRM – TOM =.168

2

0

1

0

2

0

1

1

WFI-DRM association at a site level

72.00 76.00 80.00 84.00

WFI

40.00

50.00

60.00

70.00

80.00

DR

M

2010 – DRM / WFI - 2011

WFI-DRM association at a site level

2011 – DRM/TOM

Wraparound Fidelity Index

Massachusetts Overall

WFI-4 Total Fidelity ScoresAll CSAs

Total Phase Principle

Mean Over-all

Eng Pln Imp Tr FVC TB NS Col CB CC Indiv

SB Per

OB

Mass 2010

78 86 82 79 64 88 84 53 89 74 95 70 83 83 63

Mass 2011

77 85 82 77 64 85 82 52 89 74 93 73 79 78 65

Nat Mean (CG only)

74 75 74 79 66 82 71 61 84 69 90 64 82 79 64

2010 N = 637 / 2011 N = 587

WFI Fidelity by PhaseAll CSAs 2010 N=637 / 2011 N = 587

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

2010 78% 86% 82% 79% 64%

2011 77% 85% 82% 77% 64%

Nat Mean 74% 75% 74% 79% 66%

Total Eng Plan Impl Trans

WFI Fidelity by PrincipleAll CSAs 2010 N=637 / 2011 N = 587

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2010 78% 88% 84% 53% 89% 74% 95% 70% 83% 83% 63%

2011 77% 85% 82% 52% 89% 74% 93% 73% 79% 78% 65%

Nat Mean 74% 82% 71% 61% 84% 69% 90% 64% 82% 79% 64%

Total FVC TB NS COL CB CC IND SB PER OB

Item Scores: Engagement All CSAs

ITEMS 2010 2011 NAT MEAN

CG1.1 - When you first met ICC, were you given time to talk about strengths *and* Did this process help you appreciate? 1.78 1.74 1.65

CG1.2 - Before your 1st team meeting, did your ICC fully explain the WA process and the choices you could make? 1.78 1.82 1.68

CG1.3 - At beginning of wrap process, did you have a chance to tell WF what things have worked in the past? 1.83 1.78 1.75

CG1.4 - Did you select the people who would be on your WA team? 1.41 1.34 0.86

CG1.5 - Is it difficult to get team members to attend team meetings when they are needed? 1.66 1.65 1.57

CG1.6 - Before your 1st wrap team meeting, did you go through a process of i.d.ing what leads to crises for child and family?

1.81 1.90 1.52

Item Scores: Planning All CSAs

ITEMS 2010 2011NAT

MEANS

CG2.1 - Did you and your team create a written plan that describes how the team will meet your child's needs? *and* Do you have a copy?

1.83 1.87 1.64

CG2.2 - Did the team develop any kind of written statement about what it is working on with your child and family? *and* Can you describe what your team mission says? 1.76 1.76 1.56

CG2.3 - Does your wrap plan include mostly professional services? .99 1.10 0.61

CG2.4 - Are the supports and services in your WA plan connected to the strengths and abilities of your child and family? 1.80 1.72 1.74

CG2.5 - Does the wrap plan include strategies for helping your child get involved w/ activities in his/her community? 1.31 1.26 1.24

CG2.6 - Are there members of your wrap team who do not have a role in implementing your plan? 1.73 1.73 1.67

Item Scores: Planning All CSAs

ITEMS2010 2011 NAT

MEAN

CG2.7 - Does your team brainstorm many strategies to address your family's needs before selecting one? 1.79 1.79 1.73

CG2.8 - Is there a crisis plan? *and* does this plan specify how to prevent crisis? 1.48 1.57 1.43

CG2.9 - Do you feel confident that, in crisis your team can keep your child in the community? 1.57 1.58 1.5

CG2.10 - Do you feel like other people on your team have higher priority than you in designing your wrap plan?

1.83 1.73 1.53

CG2.11 - During planning process, did team make enough time to understand values? *and* Is your wrap plan in tune w/ family's values?

1.89 1.84 1.73

Item Scores: Implementation All CSAs


CG3.1 - Are important decisions made about your child or family when you are not there? 1.90 1.86 1.64

CG3.2 - When your wrap team has a good idea for support, can they find resources or make it happen? 1.58 1.54 1.7

CG3.3 - Does your wrap team get your child involved w/ activities they like and do well? 1.05 0.95 1.2

CG3.4 - Does the team find ways to increase the support you get from friends & family? 1.09 1.13 1.22

CG3.5 - Do the members of your team hold each another responsible for doing their part? 1.73 1.73 1.7

CG3.6 - Is there a friend or advocate of your child or family who actively participates in wrap team? .68 .66 0.95

CG3.7 - Does your team come up w/ new ideas? *and* Does your team come w/ ideas when something's not working?

1.75 1.70 1.74

Item Scores: Implementation All CSAs

ITEMS 2010 2011 NAT MEANS

CG3.8 - Are the services and supports in your wrap difficult for you family to access? 1.61 1.66 1.54

CG3.9 - Does the team assign specific tasks to all team members at end of mtng? *and* Does team review team member's follow-through at next mtng?

1.67 1.63 1.59

CG3.10 - Do members of your team always use language you can understand? 1.96 1.94 1.93

CG3.11 - Does your team create a positive atmosphere around successes and accomplishments at each team meeting? 1.88 1.91 1.86

CG3.12 - Does your team go out of its way to make sure all members present ideas and participate in decisions? 1.82 1.84 1.67

CG3.13 - Do you think your wrap process could be discontinued before you're ready? 1.48 1.32 1.35

CG3.14 - Do all the members of your team demonstrate respect for you and your family? 1.96 1.91 1.88

CG3.15 - Does your child have the opportunity to communicate their own ideas when it comes to decisions? 1.49 1.39 1.71

Item Scores: Transition All CSAs


CG4.1 - Has your team discussed a plan for how wrap will end *and* Does your team have a plan for when? .46 0.70 0.68

CG4.2 - Has the wrap process helped your child develop friendships w/ other youth .94 .92 1.2

CG4.3 - Has the wrap process helped your child to solve their own problems? 1.09 1.02 1.3

CG4.4 - Has your team helped you and your child prepare for major transitions? 1.50 1.64 1.35

CG4.5 - After formal wrap ends, do you think the process will be able to be 're-started' if you need it? 1.82 1.80 1.61

CG4.6 - Has the wrap process helped your family to develop or strengthen relationships that will support you when WA is finished?

1.45 1.42 1.49

CG4.7 - Do you feel like you and your family will be able to succeed on its own? 1.33 1.41 1.22

CG4.8 - Will some members of your team be there to support you when formal wrap is finished? 1.60 1.40 1.65

WFI – Summary

• Overall: Significantly higher than national mean

• Overall, no change from 2010

• Relative strengths:

– Engagement, Planning phases

– Family voice and choice, team based, collaborative, Individualized

– Initial engagement, family selecting team, crisis planning, vision/mission statements, balance of formal and informal supports

• Relative weaknesses:

– Natural supports, involvement of friends/advocates, involvement in the community

– Youth voice, youth friendships, youth involvement in community

WFI – Summary of trends from 2010

• Areas of improvement from 2010:

– Crisis identification and plans

– Transition planning• including prep for transitions from wrap and within wrap

• Areas of decline:

– Family voice and choice – selecting team, perception of family influence

– Youth friends, youth voice, youth problem solving

– Perception that wrap could be discontinued before family ready

Team Observation Measure

Mass CBHI overall

TOM Total Fidelity ScoresAll CSAs 2010 N=285 / 2011 N=658

Total PrincipleOverall Mean

FVC TB NS Col CB CC Ind SB Per OB

Mass CBHI Mean 2010

N = 285

83 95 84 43 88 91 92 83 88 89 73

Mass CBHI Mean 2011

N = 658

85 94 85 51 92 91 93 86 90 92 78

National Mean

77 86 72 58 81 84 82 72 79 83 68

TOM Fidelity by PrincipleAll CSAs 2010 N=285 / 2011 N=658

83

95

84

43

88 91 92

8388 89

7377

86

72

58

81 84 82

7279

83

68

85

94

85

51

92 91 9386

90 92

78

0

10

20

30

40

50

60

70

80

90

100

Mean 2010 83 95 84 43 88 91 92 83 88 89 73

National Mean 77 86 72 58 81 84 82 72 79 83 68

Mean 2011 85 94 85 51 92 91 93 86 90 92 78

Total Score

F V & C

TB NS Col CB CC Indiv SB Per OB

TOM Item & Indicator ScoresAll CSAs

TEAM BASED 2010 2011 Nat Mean

Item 1: Team Membership and Attendance 3.10 3.09 2.39a. Parent/caregiver is a team member and present at the meeting. .99 .99 .93b. Youth (over age 9) is a team member and present at the meeting. .61 .61 .75c. Key school or other public stakeholder agency representatives are present. .61 .60 .38

Item 2: Effective Team Process 3.61 3.71 3.4a. Team meeting attendees are oriented to the wraparound process and understand the purpose of the meeting. .88 .91 .86b. The facilitator assists the team to review and prioritize family and youth needs. .95 .95 .89

c. Tasks and strategies are explicitly linked to goals.* .91 .95 .85d. Potential barriers to the nominated strategy or option are discussed and problem-solved. .86 .91 .84


COLLABORATIVE 2010 2011Nat

Mean

Item 3: Facilitator Preparation 3.50 3.68 3.36a. There is a clear agenda or outline for the meeting, which provides an understanding of the overall purpose of the meeting and the major sections of the meeting. .86 .91 .84b. The meeting follows an agenda or outline such that team members know the purpose of their activities at a given time. .84 .91 .84c. The facilitator has prepared needed documents and materials prior to the meeting. .90 .96 .86

d. A plan for the next meeting is presented, including time & date. .89 .91 .87

Item 4: Effective decision making 3.50 3.69 3.14a. Team members demonstrate consistent willingness to compromise or explore further options when there is disagreement. .96 .97 .91b. Team members reach shared agreement after having solicited information from several members or having generated several ideas. .87 .95 .81

c. The plan of care is agreed upon by all present at the meeting. .94 .98 .83d. The facilitator summarizes the content of the meeting at the end of the meeting, including next steps and responsibilities. .76 .82 .87


INDIVIDUALIZED 2010 2011Nat

Mean

Item 5: Creative Brainstorming and Options 3.11 3.16 2.46a. The team considers several different strategies for meeting each need and achieving each goal that is discussed. .83 .87 .78

b. The team considers multiple options for tasks or action steps. .81 .86 .73c. The facilitator leads a robust brainstorming process to develop multiple options to meet priority needs. .67 .69 .40

Item 6: Individualized process 3.53 3.70 3.3a. Planning includes action steps or goals for other family members, not just identified youth. .85 .92 .79b. Facilitator and team members draw from knowledge about the community to generate strategies and action steps based on unique community supports. .85 .91 .78c. Team facilitates the creation of individualized supports or services to meet the unique needs of child and/or family.* .89 .96 .83d. Youth, caregiver, & family members give their opinions about potential services, supports, or strategies; including describing what has or has not worked in the past. .93 .95 .91


NATURAL SUPPORTS 2010 2011Nat

Mean

Item 7: Natural and community supports 1.54 1.61 2.38a. Natural supports for the family are team members and are present. .27 .27 .26b. Team provides multiple opportunities for natural supports to participate in significant areas of discussion. .75 .80 .66c. Community team members and natural supports participate in decision-making. .72 .79 .66d. Community team members and natural supports have a clear role on the team.* .72 .81 .61

Item 8: Natural support plans 1.94 2.47 2.37a. Brainstorming of options and strategies include strategies to be implemented by natural and community supports. .70 .77 .66b. The plan of care represents a balance between formal services and informal supports. .45 .58 .36c. There are flexible resources available to the team to allow for creative services, supports, and strategies. .21 .58 .91


UNCONDITIONAL/PERSISTANCE 2010 2011

Nat Mean

Item 9: Team mission and plans 3.44 3.61 3.23a. The team discusses or has produced a mission/vision statement. .84 .90 .75

b. The team creates or references a plan that guides its work. .91 .96 .87

c. The team has confirmed or is creating a crisis plan. .78 .90 .77d. The team plan contains specific goals that are linked to strategies and action steps. .92 .96 .87

Item 10: Shared Responsibility3.66 3.72 3.42

a. The team explicitly assigns responsibility for action steps that define who will do what, when, and how often.* .87 .91 .81b. There is a clear understanding of who is responsible for action steps and follow up on strategies in the plan. .92 .92 .86c. Providers and agency representatives at the meeting demonstrate that they are working for the family and not there to represent a different agenda or set of interests. .96 .98 .94


CULTURAL COMPETENCE 2010 2011Nat

Mean

Item 11: Facilitation skills 3.55 3.62 2.97a. Facilitator is able to impart understanding about what the wraparound process is, how it will work for this family, and how individual team members will participate. .83 .88 .74b. Facilitator reflects, summarizes, and makes process-oriented comments. .89 .92 .77c. Facilitator is able to manage disagreement & conflict and elicit underlying interests, needs, and motivations of team members. .92 .92 .73d. Talk is well distributed across team members and each team member makes an extended or important contribution. .93 .92 .87

Item 12: Cultural and Linguistic Competence 3.76 3.86 3.56

a. The youth, caregiver, and family members are given time to talk about the family’s values, beliefs, and traditions. .87 .95 .86

b. The team demonstrates a clear and strong sense of respect for the family’s values, beliefs, and traditions. .95 .97 .88

c. Meetings and meeting materials are provided in the language the family is most comfortable with. .97 .98 .98

d. Members of the team use language the family can understand (i.e. no professional jargon/acronyms) .97 .98 .96


OUTCOMES BASED 2010 2011Nat

Mean

Item 13: Outcomes Based Process 2.88 3.06 2.83

a. The team uses objective measurement strategies. .67 .76 .68

b. The team assesses goals/strategies using measures of progress. .72 .77 .62

c. The team revises the plan if progress toward goals is not evident. .84 .88 .87

Item 14: Evaluating Progress and Success 2.99 3.15 2.62

a. The team conducts a systematic review of members’ progress on assigned action steps. .78 .84 .71

b. The facilitator checks in with the team members about their comfort and satisfaction with the team process. .74 .79 .73

c. Objective or verifiable data is used as evidence of success, progress, or lack thereof. .72 .78 .61


VOICE AND CHOICE 2010 2011Nat

Mean

Item 15: Youth and Family Voice 3.89 3.86 3.53a. The team provides extra opportunity for caregivers to speak and offer opinions, especially during decision making. .98 .98 .95b. The team provides extra opportunity for the youth to speak and offer opinions, especially during decision making. .93 .93 .91c. Caregivers, parents, and family members are afforded opportunities to speak in an open-ended way about current and past experiences and/or about hopes for the future. .98 .98 .95d. The youth is invited to speak in an open-ended way about current and past experiences and/or about hopes for the future. .96 .93 .88

Item 16: Youth and Family Choice 3.72 3.69 3.38a. The youth prioritizes life domains, goals, or needs on which he or she would like the team to work. .78 .79 .76b. The caregiver or parent prioritizes life domains goals, or needs on which he or she would like the team to work. .93 .96 .89

c. The family and youth have highest priority in decision making .97 .95 .90


STRENGTH BASED 2010 2011Nat

Mean

Item 17: Focus on Strengths 3.31 3.47 2.89

a. Team members acknowledge or list caregiver/youth strengths. .92 .95 .82b. Team builds an understanding of how youth strengths contribute to the success of team mission or goals. .78 .85 .67c. In designing strategies, team members consider and build on strengths of the youth and family. .82 .89 .72d. Facilitator and team members analyze youth & family member perspectives and stories to identify functional strengths. .78 .87 .69

Item 18: Positive Team Culture 3.7 3.69 3.44a. The team focuses on improvements or accomplishments throughout the meeting. .91 .92 .82b. The facilitator directs a process that prevents blame or excessive focus on or discussion of negative events. .97 .95 .89c. The facilitator encourages team culture by celebrating successes since the last meeting .88 .93 .78

d. There is a sense of openness and trust among team members. .94 .93 .90


COMMUNITY BASED 2010 2011Nat

Mean

Item 19: Community Focus 3.41 3.45 3.16a. The team is actively brainstorming and facilitating community activities for the youth and family.* .82 .85 .71

b. The team prioritizes services that are community-based. .82 .86 .79c. The team prioritizes access to services that are easily accessible to the youth and family. .93 .94 .90

Item 20: Least Restrictive Environment 3.92 3.86 3.59a. The team’s mission and/or identified needs support the youth’s integration into the least restrictive residential and educational environments possible.* .99 .97 .98

b. When residential placements are discussed, team chooses community placements for the child or youth rather than out-of-community placements, wherever possible. .87 .88 .94c. Serious challenges are discussed in terms of finding solutions, not placement in more restrictive residential or educational environments. .95 .94 .94

TOM – Summary

• Overall: Significantly higher than national mean

• Significant improvement from 2010

• Relative strengths:

– Nearly all principles. Particular strengths in:• School/Agency reps present, effective teams and facilitation, facilitator

prep, Creative/individualized services, services for all family members, crisis plans

• Relative weaknesses:

– Youth involvement/presence at meetings

– Natural supports present and involved

– Flex resources available and used when needed

TOM – Summary of trends from 2010

• Areas of improvement from 2010:

– Facilitator prep, team process

– Flexible resources available

– Balance of informal/formal services/supports

– Crisis planning and plans

– Use of objective measures and following up

– Tying strengths to planning

• Areas of decline:

– Youths being asked their priorities

– Youths present

Document Review Measure

Mass CBHI overall

Mass-DRM Principle Scores

Total Mass DRM improvement: 61% to 74%

DRM Item scores: Relative strengths 2010 2011

8 The family is present at all meetings. 2.83 2.914 There is a clearly articulated long range vision of the youth and family. 2.31 2.9

17There is evidence that the family and/or youth are making decisions about the direction and methods of the team. 2.53 2.84

11There is evidence that all members on the CPT are in regular communication, to ensure coordinated efforts on the youth and family's behalf. 2.71 2.74

15Services, supports and tasks are based on the needs of the youth and family, not availability of services or support that are needed. 2.34 2.73

23The RMSP identifies triggers, signs or behaviors that indicate the crisis is beginning and interventions to deescalate crisis situations. 2.62 2.68

2 The family and youth's needs or concerns are identified across life domains. 2.13 2.62

22The risk management/safety plan (RMSP) includes strengths of youth and family and strategies for preventing potential crises. 2.51 2.59

12There is evidence that the wraparound plan, including the RMSP, is updated at each team meeting or as needed. 2.25 2.56

18 There is documentation that progress toward goals and tasks has been monitored. 2.29 2.56

5Detailed examples of family and youth strengths, assets, resources, culture, values, and beliefs are included to support planning for areas of priority need. 2.07 2.52

25The RMSP identifies specific steps to be taken if the crisis occurs, people responsible for each step, including the youth's local ESP/MCI. 2.43 2.39

16 There is evidence that the team brainstorms more than 6 options to address goals. 1.99 2.33

DRM item scores: Needs for improvement

2010 2011

10

There is evidence that natural supports are actively involved in the planning and implementation process or there are ongoing and persistent efforts to identify and engage natural supports. 0.8 0.97

13

There is documentation that an attempt has been made to collaborate with the primary care physician (PCP) or primary care clinician (PCC).

1.72 1.21

20The wraparound plan includes opportunities for the youth to engage in community activities that he or she likes and does well. 1.05 1.34

21

There is documentation of transition and sustainability planning that identifies needs, services and support that will continue after formal wraparound ends or when the youth transitions to the adult service system. 1.16 1.4

26 The RMSP is proactively shared with the MCI team. 1.43 1.97

24The RMSP is updated as needed; always when a hub-dependent support joins the youth and family team. 1.94 2.09

6There is evidence that a strengths, needs, and culture discovery process was completed within 10 days of enrollment of the youth. 1.64 2.11

7The initial wraparound plan was developed within 35 days of enrollment with the youth and family. 2.47 2.16

14The plan includes strengths-based goals that are measurable with specific measurement strategies. 1.75 2.17

DRM Summary

• Greater correlation with WFI/TOM from 2010

• Large improvement from 2010, particularly:

– Vision statements

– Updated plans

– Needs ID’d across domains

– Examples of strengths, beliefs, culture

– SNCD within 10 days

• Declines in:

– Wrap plans w/in 35 days

– Collaboration with PCPs

DRM Psychometrics

• Total Alpha = .85

• Principal Components Analysis (w/Varimax Rotation) found Interpretable 8-Factor Solution explaining 60.6% of variance

• Correlation with WFI = .326 (p<.05)

– Certain items negatively correlate with WFI and/or have negligible correlation with DRM total

6 “Bad” DRM itemsDo not correlate with the WFI and have low item-total correlations

There is evidence that a CANS and full assessment were completed with the youth and family within 10 days of enrollment.The family is present at all meetings.

There is evidence that natural supports are actively involved in the planning and implementation process or there are ongoing and persistent efforts to identify and engage natural supports.

There is evidence that all members of the CPT are in regular communication, to ensure coordinated efforts on the youth and family's behalf.

There is documentation that an attempt has been made to collaborate with the primary care physician (PCP) or primary care clinician (PCC).The RMSP is proactively shared with the MCI team.

11 “Good” DRM ItemsSignificant correlation with WFI AND high item totals

Documentation is strengths based and identifies and prioritizes the needs for the youth and family.Detailed examples of family and youth strengths, assets, culture, values and resources are included to support planning for areas of priority need.The initial ICP was developed within 28 days of enrollment with the youth and family.The ICP specifies the goals that will help the family reach their long-range vision.There is evidence that the ICP is updated at each team meeting and as needed.Services, supports and tasks are based on the needs of the youth and family and not on the availability of services or support that are needed.There is evidence that the CPT brainstorms more than six options to address goals.There is documentation that progress toward goals and tasks has been monitored.There is evidence that the youth and family are making progress toward their goals.The RMSP identifies triggers, signs or behaviors that indicate the crisis is beginning and notes interventions to deescalate crisis situations.The RMSP identifies specific steps to be taken if the crisis occurs, people responsible for each step, including the youth's local ESP/MCI.

DRM Principal Components Analysis

• Goals and Strategies (5 items)

• Strengths and Needs (4 items)

• Progress monitoring and timely updates (4 items)

• Crisis and safety plans present and of quality (3 items)

• Natural supports and community based services evident (3 items)

• Families as primary decision makers (3 items)

• Meeting mandated timelines (2 items: CANS and ICP)

• RMSP is shared with MCI team (1 item)

DRM Principal Components Analysis

ItemStr and Needs

Goals and Strategies

Progress monitoring and timely updates

Crisis and safety

Natural supports and community based

Family de cision making

MCI sharing

Meet mandated timelines

There is evidence that a CANS and full assessment were completed with the youth and family within 10 days of enrollment. 0.098 -0.035 0.100 -0.075 0.055 0.040 0.069 0.830The family and youth's needs or concerns are identified across life domains. 0.840 0.066 0.042 -0.011 0.086 0.072 0.030 0.128Documentation is strengths based and identifies and prioritizes the needs for the youth and family. 0.745 0.125 0.135 0.156 -0.057 0.104 -0.080 -0.060There is a clearly articulated long range vision of the youth and family. 0.245 0.338 0.067 0.183 -0.237 0.435 -0.187 -0.002Detailed examples of family and youth strengths, assets, culture, values and resources are included to support planning for areas of priority need. 0.708 0.187 0.048 0.176 0.137 0.003 0.089 -0.051There is evidence that a strengths, needs, and culture discovery process was completed before the first CPT meeting. 0.614 -0.130 0.137 -0.124 0.176 0.042 0.195 0.359The initial ICP was developed within 28 days of enrollment with the youth and family. 0.026 0.214 0.110 0.208 -0.048 -0.076 -0.073 0.677The family is present at all meetings. 0.079 0.027 0.133 0.024 -0.004 0.776 0.145 0.022The ICP specifies the goals that will help the family reach their long-range vision. 0.169 0.675 0.386 -0.019 -0.051 0.098 0.006 0.020There is evidence that natural supports are actively involved in the planning and implementation process or there are ongoing and persistent efforts to identify and engage natural supports. 0.125 -0.027 0.064 0.085 0.601 0.145 -0.271 0.155There is evidence that all members of the CPT are in regular communication, to ensure coordinated efforts on the youth and family's behalf. 0.088 0.478 0.074 0.141 0.165 0.301 -0.078 -0.067There is evidence that the ICP is updated at each team meeting and as needed. 0.091 0.209 0.680 0.044 0.064 0.232 0.058 0.161There is documentation that an attempt has been made to collaborate with the primary care physician (PCP) or primary care clinician (PCC). 0.027 0.242 0.011 -0.155 -0.037 0.088 0.703 0.162The ICP includes strengths-based goals that are measurable with specific measurement strategies. 0.049 0.676 0.020 0.050 0.101 -0.035 0.319 0.085Services, supports and tasks are based on the needs of the youth and family and not on the availability of services or support that are needed. 0.253 0.600 0.403 0.018 -0.031 0.013 0.103 0.067There is evidence that the CPT brainstorms more than six options to address goals. -0.066 0.566 0.037 0.083 0.244 0.233 0.046 0.095There is evidence that the family and/or youth are making decisions about the direction and methods of the team. 0.040 0.226 0.128 0.108 0.172 0.754 0.063 -0.045There is documentation that progress toward goals and tasks has been monitored. 0.047 0.161 0.739 0.141 0.219 0.198 -0.003 0.058There is evidence that the youth and family are making progress toward their goals. 0.106 0.143 0.648 0.106 0.405 0.101 0.017 0.079The ICP includes opportunities for the youth to engage in community activities that he or she likes and does well. 0.080 0.183 0.174 -0.028 0.705 0.001 -0.051 -0.073There is documentation of transition and sustainability planning that identifies needs, services and supports that will continue after formal wraparound ends or when the youth transitions to the adult service system. 0.066 0.114 0.078 0.190 0.667 -0.014 0.390 -0.032The risk management/safety plan (RMSP) includes strengths of youth and family and strategies for preventing potential crises. 0.145 -0.019 0.204 0.777 -0.008 -0.011 0.002 0.044The RMSP identifies triggers, signs or behaviors that indicate the crisis is beginning and notes interventions to deescalate crisis situations. 0.105 0.070 0.160 0.800 0.048 0.026 0.049 0.015The RMSP is updated as needed, and reviewed at every CPT meeting. 0.119 0.029 0.538 0.222 -0.145 -0.213 0.316 0.029The RMSP identifies specific steps to be taken if the crisis occurs, people responsible for each step, including the youth's local ESP/MCI. -0.051 0.169 -0.077 0.689 0.189 0.291 -0.018 0.038The RMSP is proactively shared with the MCI team. 0.077 -0.001 0.147 0.137 -0.014 0.091 0.745 -0.109

Summary of Findings

Summary of findings

• Continued strong fidelity results, especially for a large statewide initiative

– WFI 3 points higher than national mean

• WFI scores stay stable, TOM shows small improvement, DRM show large improvement

• Big improvements noted for:

– Crisis planning

– Transition planning

– Team process / Facilitator prep

– Availability of flexible resources (but is this an artifact of TOM scoring clarification?)

– Documentation of strengths, needs, goals, crisis plans

Summary of findings

• Areas of strength continue to be:

– Managing the details of the wrap process BUT ALSO

– Family voice and choice

– Collaborative teaming

– Individualized plans with balance of formal svcs and informal supports

Looking deeper: Relative weaknesses

• Certain bigger ideas of wraparound not yet happening consistently

– Natural supports and natural support plans

– Connection to community activities and informal supports

– Full youth engagement

– Youth and family self-reliance and self-efficacy

– Youths present at meetings

• Better on some things, but still room for more improvement:

– Robust brainstorming

– Objective goal setting and monitoring of progress


• Structural barriers:

– Flexible funds

– Local community collaboratives

– Time to develop more creative plans linked to community supports and natural supports

– Consistent understanding of how FPs and ICCs work together

Possible strategies: System level

• Fiscal supports

– Use of case rates

– Availability of flexible funds

– Methods to ensure less emphasis on productivity vs fidelity and outcomes?

• Community supports, collaboration, transition planning:

– Encouragement to taking a community-level system of care approach

– Hire resource development specialist(s)

– Training and support to community collaborative teams

– Continued social marketing of CBHI and cross agency training

– Collecting info on and reviewing barriers faced by teams and CSAs

• Youth engagement

– Use of youth peer to peer support partners?

Possible strategies: Organizational level

• Natural supports:

– Specific strategies (family finding?), training, time, supervision emphasis

– Role for family partner?

• Transition planning

– Clear understanding of interactions between medical necessity determination and Unconditional Care

• Ensure clear expectations are delivered to families about transition

• Outcomes based:

– Greater attention to consistent tracking of goal attainment and meeting needs

• Use a dashboard approach?

• Staff turnover/attrition:

– Training and guidance on staff recruitment and selection; compensation, caseloads, expectations (e.g., taking vacation to attend trainings)

Possible strategies: Practice level

• Invest in specific strategies for engaging youth

– E.g., Achieve My Plan! (AMP) from Portland State University

• Individualized care

– More attention to developing individualized, creative plans in supervision, QI

• Emphasis on training and supervision on:


– Individualized care plans

– Natural support action steps

– Teaching self-efficacy

Measurement issues

• WFI: Start to interview youths

– As a way of modeling a new era of engagement of Youth

– To find out what is going on from their perspective

• How to use the TOM?

– Ceiling effect now evident

– Doesn’t correlate with WFI in Mass• Bias among raters/supervisors

– Recommend use of independent TOM raters for a subsample of selected teams

Measurement issues

• Continue to use the DRM?

– Now clearly demonstrating utility as a fidelity tool

• Could be evidence for the benefit of its continued use • OR…. Could mean don’t need to use it – duplicative

– Perhaps revise based on data and disseminate for use by supervisors / CSA QA?

Measurement issues

• Analytic questions: What influences fidelity?

– Established vs. new sites

– Caseload sizes

– Urban vs. rural

– Characteristics of families served

– Time enrolled in wraparound

– Agency culture

– Turnover

– Perception of Community supports

WHAT ARE THE OUTCOMES???

• Residential

• Medicaid expenditures/Fiscal

• CANS/Functional

Part 4: Massachusetts Data Part 4: Massachusetts Data by CSAby CSA

WFI CSA Results

0.86

0.82

0.82

0.82

0.82

0.81

0.81

0.81

0.81

0.8

0.77

0.7 0.75 0.8 0.85 0.9

18 Lawrence

29 Springfield

24 Park Street

20 Lynn

12 Gandara

2 Attleboro

23 N Central

22 New Bedford

5 Cape Ann

28 S Central

Nat Mean

CS

A

Percent Fidelity

CSA N Total Score

18 Lawrence 18 0.8629 Springfield 20 0.8224 Park Street

14 0.8220 Lynn 19 0.8212 Gandara 14 0.822 Attleboro 18 0.8123 N. Central 22 0.8122 New Bedford

21 0.815 Cape Ann 20 0.8128 S. Central 17 0.80

WFI CSA Results

0.8

0.8

0.79

0.79

0.79

0.78

0.78

0.77

0.77

0.76

0.76

0.77

0.74 0.76 0.78 0.8 0.82

10 Fall River

16 Holyoke

6 C and I

1 Arlington

25 Pittsfield

19 Lowell

32 Worcester W

8 Coastal

14 Harbor

31 Worcester E

11 Framingham

Nat Mean

CS

A

Percent Fidelity

CSA N Total Score

10 Fall River 19 0.80

16 Holyoke 16 0.80

6 Cape and Isl 19 0.79

1 Arlington 18 0.79

25 Pittsfield 21 0.79

19 Lowell 20 0.78

32 Worcester W 20 0.78

8 Coastal 21 0.77

14 Harbor 18 0.77

31 Worcester E 18 0.76

11 Framingham 20 0.76

WFI CSA Results

0.76

0.76

0.75

0.75

0.74

0.74

0.73

0.72

0.7

0.69

0.55

0.77

0 0.5 1

13 Greenfield

27 RVW

9 Dimock

21 Malden

15 Haverhill

7 CSR

3 Brockton

26 Plymouth

4 Cambridge

17 Hyde Park

30 Walden

Nat Mean

CS

A

Percent Fidelity

CSA N Total Score

13 Greenfield 19 0.76

27 RVW 20 0.76

9 Dimock 20 0.75

21 Malden 22 0.75

15 Haverhill 20 0.74

7 CSR 11 0.74

3 Brockton 18 0.73

26 Plymouth 20 0.72

4 Cambridge 22 0.70

17 Hyde Park 20 0.69

30 Walden 3 0.55

TOM CSA Results

0.93

0.93

0.93

0.92

0.92

0.91

0.91

0.91

0.91

0.9

0.77

0.7

8 Coastal

7 CSR

10 Fall River

5 Cape Ann

30 Walden

21 Malden

11 Framingham

26 Plymouth

22 New Bedford

20 Lynn

Nat Mean

CS

A

Percent Fidelity

CSA N Total Score

8 Coastal 14 0.93

7 CSR 58 0.93


5 Cape Ann 38 0.92

30 Walden 4 0.92

21 Malden 44 0.91

11 Framingham

22 0.91

26 Plymouth 18 0.91

22 New Bedford

23 0.91

20 Lynn 19 0.90

TOM CSA Results

0.89

0.86

0.86

0.86

0.86

0.85

0.85

0.85

0.85

0.85

0.83

0.77

0.7 0.75 0.8 0.85 0.9

6 C and I

16 Holyoke

9 Dimock

2 Attleboro

31 Worcester E

18 Lawrence

1 Arlington

23 N Central

4 Cambridge

15 Haverhill

13 Greenfield

Nat Mean

CS

A

Percent Fidelity

CSA N Total Score

6 C and I 32 0.89

16 Holyoke 14 0.86

9 Dimock 13 0.86

2 Attleboro 10 0.86

31 Worcester E 23 0.86

18 Lawrence 10 0.85

1 Arlington 18 0.85

23 N Central 31 0.85

4 Cambridge 16 0.85



TOM CSA Results

0.83

0.83

0.82

0.81

0.79

0.79

0.79

0.77

0.77

0.75

0.73

0.77

0.65 0.7 0.75 0.8 0.85

28 S Central

17 Hyde Park

25 Pittsfield

3 Brockton

14 Harbor

27 RVW

19 Lowell

32 Worcester W

29 Springfield

24 Park Street

12 Gandara

Nat Mean

CS

A

Percent Fidelity

CSA N Total Score

28 S Central 14 0.83

17 Hyde Park 8 0.83


3 Brockton 11 0.81

14 Harbor 18 0.79

27 RVW 25 0.79

19 Lowell 16 0.79


29 Springfield 31 0.77

24 Park Street 18 0.75

12 Gandara 6 0.73

DRM CSA Results

0.82

0.82

0.82

0.8

0.8

0.8

0.8

0.79

0.79

0.78

0.74

0.7 0.75 0.8 0.85

28 S Central

6 C and I

11 Framingham

20 Lynn

18 Lawrence

14 Harbor

31 Worcester E

4 Cambridge

1 Arlington

5 Cape Ann

MA Mean

CS

A

Percent Fidelity

CSA N Total Score

28 S Central 10 0.82

6 Cape and Isl 10 0.82

11 Framingham

10 0.82

20 Lynn 10 0.80

18 Lawrence 10 0.80

14 Harbor 10 0.80

31 Worcester E

10 0.80

4 Cambridge 10 0.79

1 Arlington 10 0.79

5 Cape Ann 10 0.78

DRM CSA Results

0.76

0.76

0.76

0.76

0.76

0.75

0.74

0.74

0.73

0.73

0.73

0.74

0.7 0.72 0.74 0.76 0.78

32 Worcester W

8 Coastal

3 Brockton

17 Hyde Park

23 N Central

10 Fall River

26 Plymouth

22 New Bedford

29 Springfield

2 Attleboro

7 CSR

MA Mean

CS

A

Percent Fidelity

CSA N Total Score


8 Coastal 10 0.76

3 Brockton 10 0.76

17 Hyde Park 10 0.76

23 N Central 10 0.76


26 Plymouth 10 0.74

22 New Bedford 10 0.74

29 Springfield 10 0.73

2 Attleboro 10 0.73

7 CSR 10 0.73

DRM CSA Results

0.71

0.71

0.71

0.7

0.69

0.69

0.69

0.66

0.65

0.65

0.64

0.74

0.55 0.6 0.65 0.7 0.75

13 Greenfield

21 Malden

16 Holyoke

24 Park Street

15 Haverhill

25 Pittsfield

27 RVW

30 Walden

9 Dimock

19 Lowell

12 Gandara

MA Mean

CS

A

Percent Fidelity

CSA N Total Score


21 Malden 10 0.71

16 Holyoke 10 0.71

24 Park Street 10 0.70



27 RVW 10 0.69

30 Walden 10 0.66

9 Dimock 10 0.65

19 Lowell 10 0.65

12 Gandara 10 0.64

Combined Scoresby CSA

0.4

0.5

0.6

0.7

0.8

0.9

1

WFI 0.82 0.86 0.81 0.79 0.76 0.8 0.77 0.8 0.81 0.79 0.77

TOM 0.9 0.85 0.92 0.89 0.91 0.93 0.93 0.83 0.91 0.85 0.77

DRM 0.8 0.8 0.78 0.82 0.82 0.75 0.76 0.82 0.74 0.79 0.74

20 Lynn18

Lawrence

5 Cape Ann

6 C and I11

Framingham

10 Fall River

8 Coastal28 S

Central22 New Bedford

1 Arlington

Nat Mean


0.4

0.5

0.6

0.7

0.8

0.9

1

WFI 0.76 0.81 0.81 0.74 0.72 0.75 0.8 0.77 0.7 0.82 0.78 0.77

TOM 0.86 0.85 0.86 0.93 0.91 0.91 0.86 0.79 0.85 0.77 0.77 0.77

DRM 0.8 0.76 0.73 0.73 0.74 0.71 0.71 0.8 0.79 0.73 0.76 0.74

31 Worcest

er E

23 N Central

2 Attlebor

o7 CSR

26 Plymout

h

21 Malden

16 Holyoke

14 Harbor

4 Cambrid

ge

29 Springfi

eld

32 Worcest

er W

Nat Mean


0.4

0.5

0.6

0.7

0.8

0.9

1

WFI 0.76 0.73 0.79 0.74 0.69 0.82 0.75 0.76 0.78 0.82 0.55 0.77

TOM 0.83 0.81 0.82 0.85 0.83 0.75 0.86 0.79 0.79 0.73 0.92 0.77

DRM 0.71 0.76 0.69 0.69 0.76 0.7 0.65 0.69 0.65 0.64 0.66 0.74

13 Greenfie

ld

3 Brockto

n

25 Pittsfiel

d

15 Haverhil

l

17 Hyde Park

24 Park Street

9 Dimock

27 RVW19

Lowell12

Gandara30

WaldenNat

mean

Percent improvement by CSA by tool

• WFI top 3 +

– 19 Lowell +.07

– 18 Lawrence +.05

– 25 Pittsfield and 22 New Bedford +.05

• WFI bottom 3 -

– 4 Cambridge -.09

– 17 Hyde Park -.14

– 30 Walden -.24 *only N = 3 in 2010


• TOM top 3 +

– 9 Dimcock +.34

– 8 Coastal + .12

– 5 Cape Ann + .12

– 25 Pittsfield +.12

– 6 C and I +.09

• TOM bottom 3 -

– 14 Harbor -.06

– 29 Springfield -.10

– 24 Park Street -.11


• DRM top 3 +

– 31 Worcester E +.43

– 32 Worcester W + .40

– 5 N. Cape Ann + .38

• DRM bottom 3 -

– 27 RVW -.10

– 19 Lowell -.13

– 30 Walden -.16

wraparound fidelity monitoring in the massachusetts cbhi summary of 2011 results august 12, 2011

Documents

family visiondevelop

family member

care planning team cpt

plan implementationbased

written plan of care

wraparound fidelity

wraparound fidelity

provisional safety plan