hb 1451 collaborative management program year 2 statewide
TRANSCRIPT
NOVEMBER 2011
FOR FURTHER INFORMATION, PLEASE CONTACT
LAURA LANDRY
303-839-9422, EXT. 170 [email protected]
ERIN INGOLDSBY
303-839-9422, EXT. 116 [email protected]
HB 1451 Collaborative Management Program Year 2 Statewide Evaluation Findings
Our Mission is to Design and Deliver Quality Human Services that Improve the Safety and Independence of the People of Colorado
November8,2011TO: LloydMalone JulieKrow ReggieBichaFROM:CollaborativeManagementProgram(CMP)EvaluationSubcommittee
OnbehalfoftheCollaborativeManagementProgram’sEvaluationSubcommittee,wearepleasedtosubmitthe2011CMPEvaluationReport.TheEvaluationSubcommitteeiscomposedofrepresentativesfromCMPcounties,stateagencypartners,andafamily‐drivenorganization.Ourworkoverthelast16monthshasbeenreviewedandapprovedbytheCMPStateSteeringCommittee.Whilethereportcontainsagreatdealofinformationontheperformanceoftheinitiative,wewantedtotakethisopportunitytothanktheStateofColoradoandtheDepartmentofHumanServices,inparticular,forongoingsupportoftheProgram.
Asyouarewellaware,thecurrentstructureofsiloedstatefundingresultsinsignificantservicefragmentationacrossagencies.Thechallengesofprovidingintegratedcarearefurthercomplicatedbydatasharingissues,conveningstafffrommultipleagencies,andbarrierstojointserviceplanninganddelivery.TheCollaborativeManagementProgramhasprovidedColoradocommunitiesacriticalframeworkbywhichtoexplore,investinandimproveservicedeliveryprocessesandinfrastructurewithinlocalsystems.Asevidencedinthefollowingreport,thishasledtomoreefficient,effectiveandmoreelegantservicedeliverywhichishavingprofoundeffectsonthelivesofColoradofamilies.
Astheinitiativematures,weareexcitedtoimplementframeworksthatwillallowustomanageprocessesmoreeffectively,disseminatelessonslearned,furtherimplementevidence‐basedservicemodels,andevaluateoureffortswithgreaterprecision.Werecognizethatongoingbudgetissuesmeanthatdifficultchoiceswillneedtobemade.WehopethatthestatecontinuestoseethepromiseoftheCMPinitiativeinbreakingdownsilos,givingvoicetoinvolvedfamilies,andimprovingoutcomesforchildrenandfamiliesinvolvedinmulti‐systemcare.
WelookforwardtobeingastrongpartnerinadvancingyourvisionforCDHSandColoradofamilies.
Sincerely,
NormanKirschAdministrator,CollaborativeManagementProgramColoradoDepartmentofHumanServices
CHILDREN, YOUTH AND FAMILIES DIVISION OF CHILD WELFARE Lloyd D. Malone, Director 1575 Sherman Street Denver, Colorado 80203-1714 Phone 303-866-4365 FAX 303-866-5563 www.cdhs.state.co.us
John W. HickenlooperGovernor
Reggie BichaExecutive Director
STATE OF COLORADO
EvaluationSubcommitteeMembersJimAdams‐BergerOMNIInstituteJustonCooperJeffersonCountyCMPSusanCollingColoradoStateCourtAdministratorsOfficeMargieGrimsleyFederationofFamiliesforChildren’sMentalHealthHeatherHarcourtElbertandLincolnCountyCMPErinIngoldsbyOMNIInstituteNormKirschColoradoDepartmentofHumanServicesLauraLandryOMNIInstituteRachelLambertJeffersonCountyCMPCarlLarsonUniversityofDenverRebeccaLarsonOMNIInstituteKamaLinscomeColoradoDepartmentofEducationKellySchrammWeldCountyCMPMegWilliamsDivisionofCriminalJustice
TableofContentsExecutive Summary.................................................................................................................................................i The Collaborative Management Program (CMP) initiative.........................................................................1
A. The CMP state initiative...........................................................................................................................................2 B. CMP statewide program evaluation......................................................................................................................4
Legislative Goal I: Develop a more uniform system of collaborative management that includes the input, expertise, and active participation of parent advocacy or family advocacy organizations.......9
A. Mandated components of CMPs............................................................................................................................9 B. Local adaptation........................................................................................................................................................10 C. Effectiveness of the interagency collaboration established by CMPs......................................................15 D. Inclusion of the input, expertise, and active participation of parent or family advocacy organizations........................................................................................................................................................................21
Legislative Goal II: Reduce duplication and eliminate fragmentation of services provided to children or families who would benefit from integrated multi-agency services....................................28
A. Coordination of client consents and assessment.............................................................................................28 B. Integration of service delivery planning............................................................................................................29 C. Coordination of payment for and provision of services to CMP youth...................................................29 D. Information sharing practices................................................................................................................................30 E. Effectiveness of CMPs in reducing duplication and eliminating fragmentation..................................33 F. Challenges associated with duplication and fragmentation.........................................................................33
Legislative Goal III: Increase the quality, appropriateness, and effectiveness of services delivered to children or families who would benefit from integrated multi-agency services...............................35
A. Service quality...........................................................................................................................................................35 B. Appropriateness of services...................................................................................................................................37 C. Effectiveness of services.........................................................................................................................................38
Legislative Goal IV: Encourage cost sharing among service providers that leads to cost-reduction for the services provided to children and families in the child welfare system, including the foster care system, in the state of Colorado...............................................................................................................40
A. Cost-sharing approaches.........................................................................................................................................40 B. Cost reductions, potential cost savings, and reinvestment of funds..........................................................42 C. Challenges related to measuring costs and benefits at the initiative level..............................................43
Legislative Goal V: Lead to better outcomes and cost-reduction for the services provided to children and families in the child welfare system, including the foster care system, in the state of Colorado.................................................................................................................................................................46
A. Defining CMP target populations........................................................................................................................46 B. Defining CMP outcomes........................................................................................................................................47 C. Plan for measurement of standard outcomes in evaluation year 3............................................................53
Year three evaluation directions.......................................................................................................................56 Endnotes.................................................................................................................................................................59 Appendices.............................................................................................................................................................61
Appendix A. Structure of state management of the CMP......................................................................................62 Appendix B: Additional reports and products...........................................................................................................63 Appendix C. Sources of additional funding reported by CMPs.........................................................................67 Appendix D. Frequently used data sources for measurement of CMP performance goals........................68
i
ExecutiveSummaryINTRODUCTION
ThefollowingdocumentisbasedupontheCollaborativeManagementProgram’s(CMP)70‐pageYear2EvaluationFindingsReport.FundedbytheColoradostatelegislatureandadministeredbytheColoradoDepartmentofHumanServices,theevaluationhascompletedthesecondyearofamulti‐yearstatewideevaluation.ThefocusoftheevaluationisonexaminingtheeffectivenessofCMPeffortsinachievingthelegislativegoalssetoutbyHouseBill04‐1451(HB1451).Theevaluationdesignisbeingimplementedinmultiplephases,includingformativeevaluationtodescribeandinformcurrentpractices,infrastructuredevelopmenttolaythefoundationforstandarddatacollection,andsummativeeffortstoassessindividualandcross‐CMPeffects.Evaluationfindingstodatehaveilluminatedawidevariationindefinedtargetpopulations,servicemodels,andmeasuredoutcomes,aslocalCMPshavetailoredtheireffortstomatchtheircommunity’sneedsandstrengths.Asdescribedbelow,individualCMPsdemonstratedprogressonkeyperformancemeasuresofeachlegislativegoal.However,thediversityofmeasurementposedsignificantchallengestoassessingstatewideimpactoftheCMP.Toaddressthesechallenges,theinitiativeundertookaprocessinthissecondevaluationyeartoidentifyindicatorsandtodesignabasicdatamanagementsystemtosupportlocalefforts.Threefutureevaluationdirectionsarerecommended:
1. Continuetodevelopandimplementsystemstocollectandanalyzewell‐defineduniformmeasuresofkeyoutcomesandprocesses,atlocalandstatelevels.
2. MonitorandtracktrendsinstatewidemeasuresofkeyCMPoutcomes.3. ContinuetodefineandsupportdisseminationofinformationoneffectiveCMPinteragency
structures,serviceprocesses,andcollaborativepractices.
CMPBACKGROUNDANDPARTICIPANTS
CollaborativeManagementProgram(CMP)InitiativeIn2004,theColoradoGeneralAssemblypassedHouseBill04‐1451basedontheassertionthat“developmentofauniformsystemofcollaborativemanagementisnecessaryforagenciesatthestateandcountylevelstoeffectivelyandefficientlycollaborate,shareresourcesormanageandintegratethetreatmentandservicesprovidedtochildrenandfamilieswhobenefitfrommulti‐agencyservices.”iThisledtothecreationoftheCollaborativeManagementProgram(CMP)whichsetsforththefollowinggoals:
1. Developamoreuniformsystemofcollaborativemanagementthatincludestheinput,expertise,andactiveparticipationofparentorfamilyadvocacyorganizations.
2. Reduceduplicationandeliminatefragmentationofservicesprovidedtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices.
3. Increasethequality,appropriateness,andeffectivenessofservicesdeliveredtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices.
4. Encouragecostsharingamongserviceproviders.5. Leadtobetteroutcomesandcost‐reductionfortheservicestochildrenandfamiliesinthechild
welfareandfostercaresysteminthestateofColorado.
ii
TheCMPinitiativewasfirstimplementedinsixcountiesin2005‐2006,withadditionalcountiesaddedannually.Intheupcomingyear,thereare30localCMPsparticipatingintheinitiative,twoofwhicharemulti‐countypartnerships(seemaponfollowingpageforparticipatingcounties.)
NumberofcountiesparticipatingintheCMPinitiative
Year 2004‐2005
2005‐2006
2006‐2007
2007‐2008
2008‐2009
2009‐2010
2010‐2011
2011‐2012
NumberofParticipatingCounties
6 12 15 24 29 31 30 32
In2006,theStateSteeringCommittee(SSC),anon‐legislatedoversightgroupcomposedofparticipatingcounties,stateagencies,andfamilymembers,determinedthatperformancebasedmeasuresshouldbedevelopedbyCountyInteragencyOversightGroups(IOG)infourareasthatmatchlegislativeintent: Childwelfare Juvenilejusticesystem Education Health/mentalhealth/otherhealthservices(e.g.,substanceabuse)
Inaddition,performance‐basedincentivedollarsaredistributedtocommunitiesbasedonademonstratedachievementofoutcomesinthesefourareasandinaccordancewithaformulaapprovedbytheStateBoardofHumanServices.
MapofparticipatingCMPcounties
iii
CMP statewide program evaluation
HouseBill08‐1005authorizedanannualexternalevaluationoftheCMPinitiative.Thelegislationrequiresthatlocalprojectsreportonthefollowing:
Thenumberofchildrenandfamiliesservedthroughtheirindividualizedserviceandsupportteamsandtheoutcomesoftheservicesprovided;
Estimatedcostsandcost‐shiftingorcost‐savingrelatedtoCMPefforts;and Informationrelevanttoimprovingthedeliveryofservicestopersonswhowouldbenefitfrom
multi‐agencyservices.
InJulyof2009,CDHShiredOMNIInstitute(OMNI)toconductthestatewideevaluationoftheCMP.Thefirstyearoftheevaluationfocusedondocumentinganddescribingtherangeoflocaleffortsandoutcomesoffundedprojects.Thesecondyearfocusedonthefollowingefforts:
1. RefinementofCMPprocessandoutcomemeasurement:AworkgroupwasformedtoselectcommonvariablesanddevelopstandardizedmeasurementprocessesfortheChildWelfareandJuvenileJusticedomains;
2. Refinementofsystemstoevaluateandshareprogressonlegislativegoals,including:
RevisionandadministrationofCMPannualreportstoimprovecontentandusability. RevisionandadministrationofCMPcollaborativeeffectivenesssurvey. Administrationandanalysisoffamilyadvocacysurveys. Refinementsandtechnicalsupportoftheweb‐basedCMPportal.
LEGISLATIVEGOALSANDEVALUATIONFINDINGSGoalI:Developamoreuniformsystemofcollaborativemanagementthatincludestheinput,expertise,andactiveparticipationofparentorfamilyadvocacyorganizations.InteragencyOversightGroupsandInteragencyServiceandSupportTeamsHouseBill04‐1451mandatedparticipationofspecificagencies,encouragedfamilyinvolvementandrequiredtheestablishmentoftwoprimarystructures:anInteragencyOversightGroup(IOG)tofacilitatecommunicationandresourcesharingandanIndividualizedServiceandSupportTeam(ISST)toplanandcoordinatethedeliveryofservicesformulti‐agencyinvolvedfamilies.Showninthetable,membershipdatareflectsubstantialparticipationbymandatedIOGpartners:
iv
CMPimpactonstaffingteams
“We’vealwayshadgoodcollaboration,butithasgonetoadifferentlevelsincebecomingapartoftheCMP.Wearemoreeffectiveinourstaffing.Weusedtooverwhelmfamilieswithtoomanypeopleintheroom.ThankstoISSTandwrap,wehaverefinedourprocess.Wehavetherightpeopleintheroomattherighttime.We’vebroughtinmorefundingopportunitiesandprogrammingasaresultofourcollaboration.Andlastly,theroleofthefamilyintheirowncaseplanningisclearerandmoreeffective.”
AgencyorIndividualsRepresentedthroughMembershiponIOG
(n=28)
CountyDepartmentofHumanandSocialServices 28 (100%) 27 (96%)
Schoolrepresentative 28 (100%) 27 (96%)
Probation 28 (100%) 27 (96%)
Mentalhealthserviceprovidercenter 27 (96%) 25 (93%)
Substanceabuseserviceprovider 27 (96%) 24 (89%)DivisionofYouthCorrections 27 (96%) 23 (85%)
Domesticviolenceserviceprovider 27 (96%) 23 (85%)
CountyHealthDepartment 26 (93%) 22 (85%)
SenateBill94representative 25 (89%) 22 (88%)
Behavioralhealthorganizationrepresentative 24 (86%) 20 (83%)Localcourts/judicial 20 (71%) 13 (65%)
Diversion 13 (46%) 12 (92%)
Familyadvocacyorganization 12 (43%) 10 (83%)
Lawenforcement 12 (43%) 7 (58%)
Familyrepresentative 9 (32%) 0 (0%)Localhealthservicesprovider 6 (21%) 6 (100%)
Electedofficial 6 (21%) 0 (0%)
Youthrepresentative 4 (14%) 4 (100%)
BusinessorChamberofCommerce 3 (11%) 2 (67%)
Other 17 (61%) 16 (94%)
NumberofCMPsReportingIOGMember(%)
NumberofCMPsReportingIOG
MemberAttendingatLeast
50%ofIOGMeetings(%)
Note:Legislativelymandatedpartnersareshowninboldtextabove.
IOGsidentifiedacommonsetofcoreactivitiesrelatedtolegislativegoals.Thefivemostcommonfocusareasinclude:
Improvingqualityofservicedeliveryforchildrenandfamilies Establishingand/orrefiningIOGstructure,processes,andprotocols Strengtheningcollaborativerelationshipswithnewandexistingpartners EnhancingCMPpersonnelresources AddressingCMPsustainability
ISSTsplanandcoordinateservicesforyouthinfamilieswiththeparticipationofmultipleagenciestodevelopanintegratedserviceplan.ThefollowinghighlightsfindingsrelatedtoISSTactivities: Onaverage,sitesreportedhavingtwoISSTs,
withsitesrangingfrom1to16.ISSTtypesincludethoseservingallfamilies,aspecializedISSTwithinonebroadservicearea(e.g.,JuvenileJustice),andthosefocusedonaspecificoutcomearea(e.g.truancyISST).
YouthandfamiliesaretypicallyreferredtoanISSTthroughaCMPpartneragency.Thetwomostcommonreferralsourcesarecounty
v
departmentsofhumanservicesandschooldistricts,respectively.Othersourcesincludedomesticviolenceproviders,primarycarephysicians(pediatricians),SenateBill94staff,andearlychildhoodeducationproviders.
Twenty‐sixCMPsindicatedthatatleastfouragenciestypicallyattendedstaffing,withthirteenindicatingparticipationofsevenormoreagencies.
Onaverage,CMPsholdfiveISSTmeetingsperfamily,thoughthisvarieswidely,fromaminimumofonetoamaximumofthirty.
TheQualityofCollaborativeProcessesOverall,IOGmembersperceivetheircollaborativestohavestrongandproductivestructuresandprocessesinplace.Onaverage,IOGmembersindicatedthattheirCMP’seffortswerecharacterizedbyhighqualityexperiencesintermsofstrongleadership,authenticprocesses,andpossessingstructuralintegrity,withsurveyscalescoresexceedingthe4.25markerofcollaborativeeffectiveness(outofa6pointscale,with6indicatinggreatereffectiveness).CMPssharedevidenceoftheeffectivenessoftheircollaborativeeffortsbydescribingkeyachievementsattheIOGandISSTlevelsinFY2010‐11,summarizedinthetablebelow:AreasofCollaborative
Effectiveness ExampleAchievementsEstablishmentofsubcommittees,leadingtomoreefficientuseofIOGmeetings•asteeringcommitteetooversee"day‐to‐day"operationalactivities•anoutcomescommitteetospecifylocalevaluationmeasuresandgoalsFullpartnerparticipationinstrategicplanningsessionsCreationofyouthadvisoryboardEstablishmentofa"virtualvoting"processwithinaruralIOGtoincreaseactiveparticipationindecision‐makingRoll‐outofnewprogramstargetingareassuchaspregnancy,mentoring,youthservicelearningIncreasesinthenumberoffamiliesreceivingspecificservicesIncreasesinthenumberofreferrals,indicatinggreatercommunityawarenessofeffectivenessofservicedeliveryDecreasedtimebetweenreferralandenrollmentininterventionservicesFundedsuccessfulgrantwritingeffortsBlendedorbraidedfundingidentifiedforspecificservicesleadingtomoreintegrationandstreamlinedservicesamongpartneragenciesEstablishmentofprocesstograntseedfundstonewprograms
ImprovedprocesseswithintheIOG
Developmentofnewprogramsor
enhancementofexistingservicedelivery
Supportingsustainability
CMPscontinuetostrugglewithsomecommonchallengesintheircollaborativeefforts.Theseinclude:
Participationandbuy‐infrompartners.CMPsexperiencedifficultyingettingparticipationfromspecificpartners,andhavingpartnersfailtofollowthroughonplans.
CMPstaffing.CMPsreportedthatinadequatestaffingandturnover(lackofacoordinator,ISSTfacilitators)hasimpactedCMPadministrationandservicedelivery.
Financialstability/sustainability.CMPsreportedchallengesassociatedwithageneralshortageoffunding,largelyattributedtotheeconomicdownturn.TheCMPhasincreasedfrom6countiesin2005‐06to31countiesin2011‐12,whiletheavailableearnedincentivefundshavenotincreased.
Informationsharing.SomeCMPshavestruggledwithestablishingpartneragreementregardingconfidentialityrequirementsandconsentprocesses/documentstoshareclientinformation.
vi
However,theColoradoChildandYouthInformationSharingcommitteewasestablishedaspartofaCMPstrategicplanandisengagedinpromotingcrosssysteminformationsharinganddatasharing(seehttp://collaboration.omni.org/sites/1451/SitePages/Home.aspxformoreinformation).
SharedCMPmission.AfewCMPs(3)citeddifficultyensuringclearandcommonunderstandingofcollaborativegoals,activities,andpracticesamongpartners,whichhasbeencompoundedbyturnoverintheIOGandpartneragencies.
FamilyParticipationThelegislationstronglyencouragesCMPstoengagefamilyrepresentativestoensurethatsystemsareresponsivetothefamiliestheyserve.FamiliescanbeinvolvedatboththeIOGandISSTlevels.
IOGlevel.Over40%ofCMPsreportedatleastonefamilymemberontheirIOG,about40%reportedatleastonefamilyadvocate,andabout20%reportedatleastoneyouthmember.
ISSTlevel.NearlyhalfofCMPs(12CMPs,46%)reportedthatfamilymembersarealwaysviewedaspartnersinserviceplanningforotherfamilies,andthevastmajority(85%)ofCMPsreportedthatfamilymembersareseeninthiswayatleast“sometimes.”
Participationinserviceplanning.Themajority(63%,17CMPs)ofCMPsindicatedinvolvingfamiliesimmediatelyafterbeingdeterminedeligibleforISSTservicesandnearlyone‐third(30%,8CMPs)reportedthatfamiliesarefirstcontactedpriortoadecisionregardingISSTeligibility.CMPs(96%)reportedthatprimarycaregiversparticipateintheISSTeitherfrequentlyoralways;themajority(77%)reportedthatimportantdecisionsaboutservicesarerarelyornevermadewithoutthefamilypresent;andthevastmajority(89%)reportedthatthefamilyfrequentlyoralwaysparticipatesintheprocessofdevelopingandwritingserviceplans,andisthengivenacopyofthatplan(93%).
CMPsidentifiedanumberofbarriersbelievedtolimiteffectiveengagementoffamiliesintheirstructuresandprocesses.ThemostfrequentlycitedbarrierwasidentifyingfamiliesappropriateforIOGparticipation(64%),followedbyalackoffamilyknowledgeorexperienceaboutCMPprocesses(39%)andconfidentialityconcerns(39%).Projectscitedeffortstorecruit,train,andretainfamilyrepresentativesaswaystoovercomethesebarriers.TheFamilyVoiceandChoiceCommitteehasbeenintegraltofacilitatinglocalefforts.
GoalII:Reduceduplicationandeliminatefragmentationofservicesprovidedtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices.CMPsreportedaddressingduplicationandfragmentationofservicedeliverybyimplementingstrategiesinfourkeyareas:1)coordinationofclientconsentsandassessments;2)integratedserviceplanning;3)arrangingpaymentforandprovisionofservices;and4)informationsharingacrossagencies.
ThemajorityofCMPs(74%)useacommonconsentform,meaningthattheindividualorfamilyneedsignonlyoneconsentdocumenttoenableintervention‐relevantinformationtobesharedacrossallagenciesinvolvedinagivencase.
vii
AbouthalfofCMPs(14)reportedusingacommonclientassessmentorsharingassessmentresultsacrossagencies(e.g.,ColoradoJuvenileRiskAssessment).
WhilenearlyallCMPsreportedISSTserviceplansasbeingintegrated(92.5%)andindividualizedtofamilyneeds(78%),onlyhalf(13)indicatedsharingasinglecomprehensiveserviceplan.Somesitesdescribedthatpartneringagenciesagreeuponservicecomponentsyetcreatemultiple(i.e.,agency‐specific)plans.
Over80%ofCMPsshareservicedeliverycostsonacase‐by‐casebasis,withindividualagenciesdesignatedtopayforparticularcomponentsofeachindividualserviceplan.
LessthanhalfofCMPshaveformaldatasharingagreeements(46%)andprocessesinplaceforpartneragenciestoaccessdatasystemsforservicedelivery(32%)orevaluationpurposes(21%).
EffectivenessofeffortstoreduceduplicationandfragmentationAssurveyresultsshow,IOGmembersperceiveCMPeffortsaseffectivelyaddressingthislegislativegoal,withaveragescalescoresexceedingthe4.25benchmark(outofa6pointscale,with6indicatinggreatereffectiveness).Despiteprogressintheseareas,thefollowingbarrierswerereportedbymultipleCMPs:
Thelackofconnectionbetweendifferentstate‐leveldatabases(e.g.,Trails,Eclipse)makesitdifficulttoobtaincompleteinformationaboutyouthbeingservedacrosssystems.Thisimpedesidentificationoftheyouthwithhighestneedforintegratedplanning.
ManyCMPsexperiencechallengesinaccessingdatainspecificsystems(e.g.,schools,privateproviders)andensuringaccessforkeypersonnel(e.g.,enablingISSTfacilitatorstoaccessTrailsdata),whichslowsdownservicedelivery.
One‐quarterofCMPshavenotyetdevelopedacommonreleaseofinformation;somecitedifficultiesinmeetingallfederalandstatelawsaswellasthepoliciesofpartneragencies.
Theeconomicdownturnhasresultedinhighstaffturnoverinpartneragencies,leadingtodelaysandgapsinservicecoordination.
GoalIII:Increasethequality,appropriateness,andeffectivenessofservicesdeliveredtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices.CMPscontinuetoidentifyandimplementservicesmeetinghighstandardsofquality,whilealsoensuringthatthoseservicesarewellmatchedtothestrengthsandneedsoftheyouthandfamiliesbeingserved.
1 2 3 4 5 6
Fewerprofessionalsservingonefamily
Morecomprehensiveserviceplans
Increasedcontinuityofservices
Greaterintegrationofservices
Streamlinedserviceprovision
Reducedduplicationofservices
viii
SeventeenCMPscitedimprovementsinservicequalityasaprimaryachievementresultingfromtheirIOGeffortsinthepastyear.NotedindicatorsincludednewlyestablishedorenhancedISSTmodels,rolloutofnewprogramstargetingservicegapsforspecifiedpopulations,processestoensureearlycasecontact,andincreasedmonitoringandfollow‐upwithfamilies.
IOGmembersindicatedstrong“commitmenttoevidence‐based/informedmodels,”ratingthissurveyitemat5outofa6pointscale.ThisisreflectedinthegrowingnumberofCMPsimplementingtheHighFidelityWraparoundmodelii(7CMPs).
Whilewidespreaduseofthesemodelsreflectspositivelyonthequalityofserviceprovision,onlyaboutone‐third(35%)ofthesesitesmeasurequalityoftheirimplementationefforts(i.e.,fidelity).
NineteenCMPssupporttheuseofotherevidence‐informedpreventiveinterventions(e.g.,Multi‐SystemicTherapy,FunctionalFamilyTherapy,LifeSkillsTraining,IncredibleYears).iii
ThemajorityofCMPsimplementedatleastoneofthestrategiesspecifiedinthetablebelowtoincreasetheappropriatenessoftheirservicedeliveryinthelastyear.
CMPstrategiestoenhanceappropriatenessofservices NumberofCMPs(%)
IdentifiedserviceareathatiscurrentlynotbeingaddressedthroughinformalneedsassessmentwithCMPpartners(n=27)
24 (89%)
IdentifiedserviceareathatiscurrentlynotbeingaddressedthroughformalneedsassessmentwithCMPpartners(n=27)
16 (59%)
Implementedanewprogram/modelthatspecificallytargetsthepopulationandserviceneed/gapthatwasidentifiedbyIOG(n=27)
20 (74%)
Implemented/enhancedservicestobemoreculturallyappropriate(n=27) 11 (41%)
AchievementsrelatedtoCMPeffortstoincreaseeffectivenessOverall,IOGmembersreportedthattheirCMPsimplementpracticesthatresultinmoreeffectiveservicedeliveryandimprovedprogramoutcomes.Averageratingsofrelevantsurveyitemsexceededthe4.25effectivenessbenchmarkoutofa6pointscale,with6indicatinggreatereffectiveness.
“BecauseoftheCMP,weareseeingsystemsbeginningtorecognizewhatisnotworkingintheirprocessandwantingtoaddressit.Insteadofpointingfingers,we’rehavingmoreconversationswithawin/winattitude.Youtharenotbeinglabeledas“YourProblem”,ratherthecommunityasawholeisembracingtheirneedscollectively.Thisisatremendousstepintherightdirection.Trustisgrowingandsystemsrealizenooneisouttotearthemdownfortheirmistakes.RathertheIOGisheretohelpeachothersucceedforthebestinterestoftheyouth.”
1 2 3 4 5 6
Improvedqualityofservices
Committedtoevidence‐based/informedmodels
Easierforfamiliestoaccessservices
Committedtoa"nowrongdoor"approach
ix
Cost‐sharing:Useofblendedandbraidedfunds
“[OurCMP]continuestoincreasethefrequencyofblended/braidedfundingforthepurchaseofspecificallyidentifiedservicesforyouth/familieswhichhasincreasedeffectivenessoftheintegratedplansforclients,aswellasincreasedthestrategicuseofmultiplefundingstreams,andreducingfragmentationofservicesbetweenagencies.Ithasalsoelevatedsomediscussionsandinformalagreementsaroundwhopaysforwhatandwhenasitrelatestoasystemicapproachratherthanagency‐specific.Thereisincreasedcooperationonprojectsbetweenpartneragencieswhenitcomestothepursuitorinvolvementingrants,whichhasalsosupportedmorecross‐systemtrainingofstaffincertainareas.”
GoalIV:Encouragecostsharingamongserviceprovidersthatleadstocost‐reductionfortheservicesprovidedtochildrenandfamiliesinthechildwelfaresystem,includingthefostercaresystem,inthestateofColorado.Thelegislationassumesthateffectiveinteragencyservicecoordination,decreasesinduplication,andpreventionofdeeperandmorecostlyfamilyinvolvementinsocialservicesystems,shouldleadtobetterfamilyoutcomesandnetcost‐savingsovertime.Testingthecost‐savingshypothesisimpliedinthislegislativegoalataninitiativeleveliscomplicatedgiventherangeanddiversityofexistingprojects,processesandoutcomes.However,localCMPshavedevelopedprocessestosharecosts.Inaddition,CMPshavesuccessfullyleveragedtheircollaborativeeffortstoobtainadditionalfundingandre‐investsavingstosustainand/orgrowtheirprogramsandservices. IOGpartnersfrequentlyoralwaysjointlypaidforstaffing(20CMPs)andadministrativecosts(25
CMPs),bypoolingfundstosupportsalariesofIOGcoordinatorsorprogrampersonnel,andcontributingin‐kindresourcessuchasofficespaceandtrainings.
Asnotedearlier,CMPsregularlysharecostsofinterventionservices.Asacomplementtocost‐sharing,seventeenCMPsreportedallocatingnearly$714,000oftheirFY2010‐11incentivefunds(approximately29%oftheirtotalfunds)topayfordirectfamilyservices.
Tenofthe29CMPs(34%)reportedreceivingcompetitivefundsfromfederalandstateagenciesandprivatefoundations.Together,theseeffortsleveragedatleast$1,084,776inadditionalfunding.ManyCMPsalsoreceivednon‐competitivefunds.
CostsavingsassociatedwithCMPeffortsThelegislationsupportsCMPsinre‐investingrealizedcostsavingsinlocalefforts.However,measurementofcostsavingsischallenging,asitrequiresacalculatedestimateofthepotentialcostsforagivenindividualorfamilyhadCMPeffortsnotbeenprovided.Inaddition,whileoutcomesmaybeimproved,itisdifficulttodeterminewhetherthesewereachievedatlowercost,andsomemayarguethateffectivelyservingmulti‐systemsinvolvedfamiliesmayactuallyincreasecosts,atleastintheshort‐term.
Only3CMPsreportedhavingaformalprocesstomeasureactualcostsavings,although12CMPsprovidedmonetaryestimatesofperceivedsavings.Examplemeasuresofsavingsinclude:reductionsintheaveragenumberandlengthoftimeCMP‐servedchildrenwereinfostercareandotherplacements;comparisonofcostsofservicesfamiliesreceivedpriorto,andfollowingCMPservices;andreduceduseofemergencyhousingassistanceandTANFfunds.
x
TheseCMPsreportedatotalof$5.9millionthatwerereinvestedinCMPefforts.CMPsutilizedthesefundsmostoftentopurchaseadditionalservicesortoservemorefamilieswithexistingservices(16CMPs).Otherfrequentlymentionedusesofreinvestedfundsincluded:contributingtocoordinatorsalaries(4CMPs),infrastructurebuilding(e.g.,databasedevelopment),family“emergency”assistance,andcontributingtolocalevaluationcosts(2‐3CMPs).
CMPsdescribedothersourcesofperceivedcostreductions,butwereunabletoquantifyassociatedcostsavings.Theseinclude:fewerfamiliesenteringthechildwelfaresystem;lowertruancyrates(leadingtopresumedlowereducationcostsandincreasedschoolrevenues);reductionsinthelengthofmentalhealthtreatmentandinpatientstays;improvementsincoordinationandaccesstoneededcareandfundingsourcessuchasMedicaidandCHP+;andpreventionofcostlyservicesthroughtargetedprograms,includingpregnancyprevention.
GoalV:Leadtobetteroutcomesandcost‐reductionforservicesprovidedtochildrenandfamiliesinthechildwelfaresystem,includingthefostercaresystem,inColorado.CMPsprovidedestimatesofeligibleandservedpopulationsandperformanceonlocally‐definedgoals,includingchildandfamilyoutcomes.
Eligiblepopulation.Currently,itisdifficulttoobtainprecisenumbers,asthereisnostandarddefinitionofeligibility,youthmayberepresentedmorethanonce(duplicatedcases),andCMPstakevaryingapproachestocalculatethetotal.Thus,thecountsarelikelyoverestimated.
o InFY2010‐11,70,000individualswereconsideredeligibleforCMPservices.Abouthalf(30,000)ofthatpopulationwasreportedlyscreenedforCMPservices.
Servedpopulation.Thetotalservedwerecalculatedatdifferentlevelsofservicedelivery(allincludeduplicatedcases):
o 19,600individualsparticipatedinlocalservicesoreffortsthatwereassociatedwiththeirCMP(e.g.,paidbyincentivefunds,donatedin‐kind,providedbypartneringagencies).
o 7,000individualsreceivedservicesthroughthelocalISSTprocess.o 5,300individualsreceivedservicesthatwerefunded,atleastinpart,byCMPincentive
funds(among19CMPswheretheirIOGfinanciallysupportedspecificprograms).BecauseCMPsemployavarietyofservicemodelsthataretargetedtoindividualanddiversecommunityneeds,localsitestendtotracktheirperformanceinmanyoutcomes,utilizingmultipleindicatorswithinoutcomes.InFY2010‐11,thereviewofindicatorsrevealed:
The28CMPswithatleastoneyearofdatatrackedacombined198performanceindicators. Two‐thirds(134)measuredpopulation‐leveloutcomes(e.g.,childandfamily),while64measured
process(e.g.,servicedeliveryimprovements).Performancewasmeasuredatdifferentlevels(i.e.,rateforcountyversusyouthservedthroughtheISST).
Onaverage,eachCMPtracked7indicators,althoughafewCMPsmeasuredtwentyormore. CommondatasourcesincludeCDHSTrails,Eclipse/ICON,schooldistrictdata,andDBH/CCAR.
xi
Intermsofperformance:
CMPsreportedmeeting78%ofthe198performanceindicatorgoals.Successinmeetinggoalswashighinthreedomains,withlesssuccessseenintheeducationdomain.
Performanceonindicatorsreflectingthemostcommonlymeasuredoutcomesissummarizedinthetablebelow.Themajorityofperformancegoalstargetingoutcomesinthechildwelfareandjuvenilejusticedomainsweremet,withmorevariableperformanceonoutcomesineducationandhealth/mentalhealthdomains.
Domain CommonOutcome %GoalsMetFY2010‐11
ChildWelfare
Enhancestabilityofout‐of‐homeplacements 88%Preventout‐of‐homeplacement/increasereunification 80%
JuvenileJustice
Successfulcompletionofprobation 69%Loworreducedusageofcommitment/detentionfacilities 86%Reduceormaintainlowratesofre‐offense/recidivism 88%
EducationIncreaseattendance/reducetruancy 44%Increaseorimprovestudentachievement 60%
Health/MentalHealth
Improvedleveloffunctioninganddecreaseinproblemseverity 60%Increasepreventionandtreatmentforsubstanceuse/abuse* 100%Decreaseormaintainlowratesofhospitalization/inpatientservices 57%
*Note:OnlyoneperformancemeasurewasassessedinthisoutcomeareainFY10‐11.
AlthoughCMPsreportedonindicatorsrelatedtocommonoutcomes,thetargetpopulationsandspecificperformanceindicatorsforthoseoutcomesweredefinedandmeasureddifferentlyacrossCMPs.Thus,datacouldnotbemeaningfullyanalyzedacrosssitestoobtainapreciseestimateofimpactintheseoutcomesacrossthestatebecausedirectcomparisonisneitherpossiblenorappropriate.PlanforcommonmeasurementacrossCMPsinevaluationyear3ThediversityofmeasuresacrossCMPsledtothespecificationofastandardmeasurementplan.InJuly2011,sitesbeganuniformmeasurementofaselectsetofprocessandoutcomeindicators,whichwillbeaggregatedatthestatelevel.TheprocessindicatorswillfacilitateanaccuratecountanddescriptionoftheCMP‐targetedpopulationandtrackprogressoncoreISSTservicecomponents.Theseinclude:
Numberofchildren/familiesreferredtoISSTs Numberandtypeofagenciesorsystemsthat:familiesareinvolvedinatthetimeofenrollment;
participateinthefamily’sserviceplanning;andprovideservicestofamiliesasaresultoftheISST Numberofcaseswherefamilymembersparticipatedinserviceplanning Numberofcasesinwhichanintegratedserviceplanwasdeveloped
Theoutcomeindicatorswillmeasurecoreexpectedoutcomesinthechildwelfareandjuvenilejusticedomains,utilizingtheTrailsandICON/Eclipsestatewidedatabases.Theseinclude:
Number(rate)ofCMP‐servedyouth:withnewopeninvolvementsinTrailsafterCMPservicesbegan,withnosubstantiatedabusefinding,dischargedtoapermanenthome;andnumberofmovesexperiencedwheninout‐of‐homeplacement
xii
Number(rate)ofCMP‐servedyouthwho:successfullycompleteprobationand/orparole;recidivate;andexperiencerevocationsbytechnicalviolations
Ameasurementtoolandonlineclientleveldatabase(ETO©)wascreatedtosupportCMPdatacollection.InYear3,indicatorsintheeducationandhealth/mentalhealthdomainswillbeselected.
FUTUREEVALUATIONDIRECTIONSInyeartwo,theevaluationcenteredonidentifyingstandarddatacollectionopportunitiesacrossCMPsandestablishingmeasurementsystems,andinformingCMPbestpracticesandprogrammodelsthroughongoingprocessandformativeevaluationefforts.YearthreeevaluationactivitiesareintendedtoincreasecapacityandenhancedataqualityinwaysthatwillallowforanexaminationofCMPoutcomes.Theshiftfromformativeevaluationandinfrastructurebuildingtoasummativedesignwillrequirelocaladjustmentsindatacollectionpracticestoensureconsistentandrigorousmeasurementstatewide,whichmaybechallengingtosomesites.OMNI,CDHS,andtheEvaluationSubcommitteehaveproposedthefollowingyearthreeevaluationactivities:1. Continuetodevelopandimplementsystemstocollectandanalyzewell‐defineduniformmeasuresof:
a. OutcomesacrossCMPsonCMP‐targetedpopulations.OMNIwillcontinuetosupportlocalCMPsintheirdatacollectionofstandardandlocaloutcomemeasures.Mechanismstomeasurestandardoutcomesineducationandhealth/mentalhealthdomainswillbecollaborativelydevelopedandimplementedinFY12‐2013.
b. Outputandprocessmeasures.OMNIwillworkwithlocalCMPstofurtherrefineanddevelopadditionalmeasurementstrategiesincollaborativeprocesses,familyinvolvement,serviceduplicationandfragmentation,quality,effectiveness,andappropriatenessofservices,andcostsandestimatesofcostsavings.Theon‐linestatewidemeasurementdatacollectionsystemprovidesastructureandmechanismbywhichlocalCMPscoulddevelopadditionaluniformmeasuresofCMPprocesses.
2. MonitorandtracktrendsinstatewidemeasuresofkeyCMPoutcomes.OMNIandtheECwilldevelopan
on‐lineplatformtodisplaydatatrendsovertimeinthefourCMPoutcomedomainsatlocalandstatelevels,whichwillraiseawarenessoftheinitiativeamongstateleadersandcommunitymembers.
3. ContinuetodefineandsupportdisseminationofinformationoneffectiveIOG,ISST,andcollaborative
practices.OMNIevaluatorsareleadingorparticipatingineffortstoreviewCMP‐relevantresearchanddisseminatebestpractices.Theseactivitiesinclude:
a. Developmentofa“SystemsFacilitationGuide”summarizingresearchandlearningonsystemscollaborationandCMPcorecomponents(familyinvolvement,integratedserviceplanning).
b. Briefevaluationandresearchreportsonkeytopics.ThesequarterlyreportswillexploretopicsrelevanttostateandlocalCMPstakeholders,suchasmeasurementofcostsavings.
xiii
c. CMPHandbookdevelopment.ThehandbookwillorientnewCMPsandnewstakeholderstotheinitiative,theestablishinglegislation,SSCpoliciesandprocedures,localevaluationandmeasurementprocesses,andcollectivelearningregardingeffectiveimplementationofIOGandISSTstructuresandprocesses.
d. CMPcommunitynetworking.OMNIwillcontinuetosupportCMPinformationsharingthroughmaintainingtheCMPSharePointportal(anon‐lineinformationsharingtool)andattendingand/orfacilitatinglocally‐initiatedregionalmeetings.
e. Familyinvolvementtechnicalassistance.OMNIwillcontinuetosupporttheFamilyVoiceandChoiceCommittee’seffortstoexamineevaluationfindingstoidentifytechnicalassistanceneedsrelatedtofamilyinvolvementacrosstheinitiative.
f. Incentivefunding.OMNIwillfacilitatetheSSC’sdiscussionsregardingthedevelopmentofnewmethodstodetermineperformancebasedincentivefundingallocations.
Movingforward,theevaluationwillfocusoncontinueddevelopmentofmeasurementinfrastructuretocreateasolidfoundationforsummativeevaluationofCMPeffortsstatewide.OMNIwillcontinuetogroundtheevaluationinaparticipatoryapproach,workinginclosecollaborationwiththeCMPStateSteeringCommitteeandtheEvaluationSub‐committee,inanefforttoensurethatresultsarewellalignedwiththeneedsandinterestsoftheinitiativeanditsstakeholders.
iColoradoRevisedStatute,Title24,Article1.9.(2010).Retrievedfromhttp://www.michie.com/colorado/lpext.dll?f=templates&fn=main‐h.htm&cp.http://www.michie.com/colorado/lpext.dll?f=templates&fn=main‐h.htm&cp.iiBrunsEJ,SatherA,StambaughL.(2008).Nationaltrendsinimplementingwraparound:ResultsfromtheStateWraparoundSurvey,2007.In:Bruns,E.J.&Walker,J.S.,(Eds.),Resourceguidetowraparound.Portland,OR:NationalWraparoundInitiative,ResearchandTrainingCenterforFamilySupportandChildren’sMentalHealth.iiiSAMHSA:APocketGuidetoEvidence‐BasedPrograms.(2011).Retrievedfromhttp://www.samhsa.gov/ebpwebguide/appendixA_Schools.asp
1
TheCollaborativeManagementProgram(CMP)initiativeIn2004,theColoradoGeneralAssemblypassedHouseBill04‐1451(referredtoasHB1451)toestablishcollaborativemanagementprogramsdesignedtoimproveoutcomesforchildrenandfamiliesinvolvedwithmultiplecountyagencies.TheGeneralAssemblydeterminedthat“developmentofauniformsystemofcollaborativemanagementisnecessaryforagenciesatthestateandcountylevelstoeffectivelyandefficientlycollaborate,shareresources,ormanageandintegratethetreatmentandservicesprovidedtochildrenandfamilieswhobenefitfrommulti‐agencyservices.”iTheresultingCollaborativeManagementProgram(CMP)isdesignedtoimproveboththequalityandcost‐effectivenessofinterventionsforColoradochildrenandfamiliesinvolvedwithmultiplegovernmentalprogramsandcommunityagenciesstemmingfromhealth,education,childwelfare,andjuvenilejusticesystemcontact.Thelegislationcallsforthedevelopmentoflocalcollaborativemanagementstructuresandprocessesthatbringtogetheragenciesandservicesforat‐risk,highsystems‐usechildrenandfamilies.PartnersinlocalCMPsincludecountydepartmentsofhuman/socialservices,localjudicialdistricts,healthdepartments,schooldistricts,communitymentalhealthcentersandBehavioralHealthOrganizations,parentorfamilyadvocacygroups,andcommunityagencies.Thespecificgoalsofthelegislationareasfollows:
1. Developamoreuniformsystemofcollaborativemanagementthatincludestheinput,expertise,andactiveparticipationofparentadvocacyorfamilyadvocacyorganizations
2. Reduceduplicationandeliminatefragmentationofservicesprovidedtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices
3. Increasethequality,appropriateness,andeffectivenessofservicesdeliveredtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices
4. Encouragecostsharingamongserviceproviders5. Leadtobetteroutcomesandcost‐reductionfortheservicesprovidedtochildren
andfamiliesinthechildwelfaresystem,includingthefostercaresystem,inthestateofColorado.
2
Thelargergoaloftheinitiativeistoimproveoutcomesformulti‐systeminvolvedyouthandfamiliesthroughcross‐systemserviceplanningandcoordination.Researchhasdemonstratedthatthesecollaborativepracticesyieldimportantbenefitsincluding:
Increasedprobabilityofimprovementinchildandfamilyoutcomes
Maximizationofavailableresourcesfortheprovisionofservices Increasedcoordinationwithinandamongservicedeliverysystems
SharedresponsibilityacrosssystemsandserviceprovidersiiImportantly,thelegislationreflectsalonghistoryofsystemreforminColoradobasedonSystemsofCareprinciples.Coreelementsincludecommunitycollaboration,familyinvolvementinserviceplanninganddelivery,andculturallycompetentservicestailoredtotheuniqueneedsofdifferentpopulations.iiiTheseelementsareusedtoengagestakeholdersoutsidestateandlocalgovernmentinconsensus‐orientedeffortstomanagepublicresourcesandcollectivelysolveproblems.Inpart,communitycollaborationhasbecomeahallmarkofsocialservicesreforminColoradoduetoresearchindicatingitseffectivenessinengagingdiversedisciplinestoaddressissuesthathavemultiplecausesandsolutions.iv
A. TheCMPstateinitiative
TheCMPinitiativewasfirstimplementedinsixcountiesin2005‐2006,withadditionalcountiesaddedannually(seeTable1below).Inthe2011‐2012fiscalyear,thereare29localCMPsparticipatingintheinitiative,twoofwhicharemulti‐countypartnerships.Table1.NumberofcountiesparticipatingintheCMPinitiative
Year2004‐2005
2005‐2006
2006‐2007
2007‐2008
2008‐2009
2009‐2010
2010‐2011
2011‐2012
NumberofParticipatingCounties 6 12 15 24 29 31 30 31
Participatingcountiesvaryinsize,populationdensity,collaborativestructure,servicedeliveryapproach,andstatedoutcomes.Figure1providesastatewidemapofparticipatingcountiesbygeographictype.
“Collaboration…hasbeenseenasameansbywhichcomplexproblemswithinterrelatedcausescanbeaddressed,astrategyformaximizingtheefficientuseoflimitedresources,awayofreducingthefragmentationwithinandbetweenbureaucracies,[and]ameansofengagingcitizensinademocraticprocessofdecision‐making.”
Emshoffetal.,2007,AmericanJournalofCommunityPsychology
3
Figure1.CMPparticipatingcountiesbypopulationdensityandyearjoined
TheinitiativeismanagedbytheColoradoDepartmentofHumanServices(CDHS),withinputfromanon‐legislatedStateSteeringCommittee(SSC),andanumberofstandingandad‐hocsubcommittees.ThestructureofthesegroupsispresentedinAppendixA.SSCCommitteemembersincludeCMPcoordinators,stateagencyrepresentatives,astatefamily‐drivenorganization,andotherkeystakeholdersandpartners.In2006,theSSCdeterminedthatperformance‐basedmeasuresshouldbedevelopedbyCountyInteragencyOversightGroups(IOG)infourareasthatmatchlegislativeintent:
Childwelfare
Juvenilejusticesystem Education Health/mentalhealth/otherhealthservices(e.g.,substanceabuse)
4
Torewardcommunityefforts,performance‐basedincentivedollarsaredistributedbasedonachievementofoutcomesinthesefourareasandinaccordancewithaformulaapprovedbytheStateBoardofHumanServices.CMPsarerequiredtosetspecificgoalsinallfouroutcomeareasandtodescribehowthesewillbemet.CMPsuseincentivefundstosupportthecostsofcollaborativeprocesses(e.g.,coordinatorsalary)andservicedelivery(e.g.,existingandnewprograms).Projectsmayalsorequestwaiversofrulesandopttoreinvestanygeneralfundsavingsintoadditionalservicestofamiliesthatwouldbenefitfrommulti‐agencyservices.
B. CMP statewide program evaluation
In2008,HouseBill08‐1005authorizedanannualexternalevaluationoftheCMPinitiative.Specifically,ColoradoRevisedStatute(24‐1.9‐102.5)statesthattheDepartmentofHumanServices,withinputfromthecounties,involvedstateagencies,participatingstakeholders,andfamilyadvocacyorganizationswilldevelopthecriteriaandcomponentsoftheexternalevaluation.Thelegislationrequiresthatlocalprojectsreportonthefollowing:
1. thenumberofchildrenandfamiliesservedthroughtheirindividualizedserviceandsupportteamsandtheoutcomesoftheservicesprovided;
2. estimatedcosts,aswellascost‐shiftingorcost‐saving,relatedtoimplementingtheCMPapproach;and
3. informationrelevanttoimprovingthedeliveryofservicestopersonswhowouldbenefitfrommulti‐agencyservices.
InJulyof2009,CDHShiredOMNIInstitute(OMNI)toconductthestatewideevaluation.OMNIisanon‐profit,socialscienceresearch,evaluationandtechnicalassistancefirmbasedinDenver,Coloradoestablishedin1976(formerlyOMNIResearchandTraining,Inc.).InOctoberof2009,OMNIbeganworkinginpartnershipwiththeSSCandtheProgramDirectoratCDHStodeveloptheevaluationdesignandanEvaluationSubcommittee(EC)wascreatedtoguideevaluationefforts.MembershipintheECisopentoCMPpartnersandcurrentmembersincludeIOGcoordinatorsfromfourcounties;theProgramDirectoratCDHS;representativesfromtheDepartmentofCriminalJusticeandtheStateJudicialBranch;representativefromastatefamily‐drivenorganization;OMNIstaff;andresearchersfromtheUniversityofDenver.i. Phasesoftheevaluation
TheoverarchingfocusoftheevaluationisonexaminingtheeffectivenessofCMPeffortsinachievingthelegislativegoalsofHB1451(e.g.,increasedfamilyinvolvement;reducedduplicationandfragmentation;increasedquality,effectivenessandappropriatenessof
5
services;greatercostandresourcesharingacrossagencies;andimprovedchildandfamilyoutcomes).Becausetheprojecthadbeenoperationalforseveralyearsbeforecommencingtheevaluation,thedesignisbeingimplementedinmultiplephases.Theseincludeformativeevaluationtoserviceanddescribecurrentpractices,infrastructuredevelopmenttolaythefoundationforstandarddatacollection,andsummativeeffortstoassessindividualandcross‐siteeffects.Eachofthesephasesisdescribedbelow.PhaseI–Formativeassessment.Formativeeffortscollectdataintendedtohelpidentifyandrefineprogramactivities.Thefocusofdatacollectionistoexamineimplementation,identifyingbarriers,observedsuccesses,andotherqualitativeinformation,inordertogainanunderstandingoftheprogram.Informationgatheredcanbeusedtodevelopstrategiestostrengthentheprogram.Formativeevaluationmethodshelptosurfaceprogramneedstobeaddressedinthesecondphaseofinfrastructurebuilding.PhaseII–Infrastructurebuilding.Thesecondphaseoftheevaluationseekstolaythefoundationforstandardevaluationpracticesacrosssites.Evaluationeffortsfocusonthedevelopmentofmeasurementstrategies,implementationofdatacollectionsystems,andbuildingthecapacityoflocalprojectstoparticipateintheevaluation.Formativeevaluationtechniquesarealsousedinthisphasetofurtherinformprogramimplementationandsupportimprovementefforts.PhaseIII‐Summative.Summativeevaluationsexaminecumulativeoutcomesofaprogramorinitiative.Inthisphase,evaluationeffortsfocusonanalyzingdatatoexaminevariationinperformanceoutcomesasafunctionofdifferencesinpracticesandprocesses.Usingthistypeofanalyticapproach,theevaluationcandetermineprojecteffectsaswellasreflectonperformanceeffortstoidentifyeffectivepracticesandopportunitiesforfurtherprogramrefinement.TheCMPstatewideevaluationiscurrentlyinphasetwo.PhaseIwascompletedinthefirstyearinordertodocumentanddescribetherangeoflocaleffortsandselectedoutcomes.Thisworkhelpedtoilluminateawidevariationindefinedtargetpopulations,servicemodels,andmeasuredoutcomes.Suchdiversityposessignificantchallengesforthespecificationandapplicationofauniformmeasurementstrategyandrelatedoutcomeanalysis.Toaddressthesechallenges,OMNIrecommendedthattheprojectdevelopuniformoutputmeasures(e.g.,clientsserved,servicesprovided),asmallsetofcommonoutcomemeasures,andabasicdatamanagementsystemtosupportlocalcollectionefforts.Thisefforttobuildevaluationinfrastructurebeganinthesecondyear(2010‐2011)oftheevaluation,withtheECandCMPstakeholdersworkingtoestablishstandardmeasurementanddatasystemstosupportfutureevaluationactivities.Infrastructurebuildingwillcontinueandsummativeevaluationeffortswillbeginintheupcomingyear.
6
Itshouldbenotedthatthemovetowardmorerigorous,cross‐sitedatacollectionprocessesthatsupportuniformmeasurementofeffortsandoutcomeshasresultedinconcernsbeingvoicedbyasubsetofCMPcommunities.Thisistobeexpectedassitesinprioryearshavebeenabletosubmitdatainwaysthatdonotconformtostandardorrigorousdatacollectionstandardsormethods,andwhichcannotbeverified.Thus,therequesttoparticipateinastatewideevaluationrequiringsomeadditionaldatacollectionmaybeperceivedbysomeasanunwelcomeburden.OMNI,theEC,CMPCoordinators,inconjunctionwiththeprojectdirector,haveworkedcarefullytodevelopadatacollectionprocessthatinvolvesaverysmallsetofvariablesforthispurpose.OMNIstaffwillcontinuetoworkwithsitestosupportcollectionefforts,whilealsocreatingdatareportingtoolsthatwillhelpcoordinatorsandlocalISSTsusecollectedinformationtosupportserviceandmanagementactivities.ii. Yeartwoevaluationmethods
Thefollowingdescriptionoutlinestheevaluationeffortemployedinyeartwo.1. RefineCMPmeasurementprocessesInOctober2010,theEvaluationSubcommittee(EC)proposedtheformationofatemporaryworkgrouptohelpselectvariablesandrefineCMPmeasurementprocesses.Accordingly,anOutcomesWorkgroup,comprisedofkeyCMPstakeholders,wasassembledtoexplorethebenefitsofimplementinguniformmeasurementprocesses;examinekeyassumptionsandintentionsofthelegislationtoselectcriticalmeasurementareas;identifyasetofcommonindicators;anddevelopastandardmeasurementapproachtobeimplementedstatewide.LedbyOMNIandwithguidancefromtheEC,theworkgroupmetthreetimesforhalf‐dayretreatsinNovember2010,January2011,andFebruary2011.ThespecifictaskscompletedbytheOutcomesWorkgroupincludedthefollowing:
Identifiedwaystoimprovemeasurementprocessesthatarebeneficialtotheinitiative,bothlocallyandatthestatelevel
ExploredkeyassumptionsoftheCMPlegislationandidentifiedevaluationandmeasurementtargetsthatarecentraltotheinitiative
DevelopedameasurementapproachtobeimplementedacrossallCMPs SelectedanddefinedprocessandoutcomeindicatorsintheChildWelfareand
JuvenileJusticedomainstobemeasuredinthe2011‐12fiscalyear(andbeganexploringaprocessforlaterselectionanddefinitionofEducationandHealth/MentalHealthindicators)
Surfacedissuesandchallengesrelatedtodatacollectionandmeasurement,andidentifiedopportunitiestoaddresstheseproblems
7
DevelopedaplantodisseminateinformationtoStateSteeringCommitteeinMarch,2011andmaderefinementsbaseduponfeedback
FacilitatedimplementationintimeforintegrationoftheplanintoMOUsforthe2011‐12fiscalyear
TheeffortsoftheOutcomesWorkgroupledtothespecificationofastatewidemeasurementplanadoptedbytheSSCandrolledoutinJuly2011.OMNIidentifiedrelevantmeasurestosupportdatacollectionanddevelopedprotocolsfordataentryintoaweb‐basedmanagementsystem.Theplanisfurtherdescribedinthefinalsectionofthisreport.2. RefineandmaintaininfrastructuretoevaluateandshareprogressonlegislativegoalsTheevaluationestablishedandcontinuedsystemstodocumentandshareCMPprogress:
RevisionandadministrationofCMPannualreports.OMNIfurtherrefinedthestructureandcontentoftheCMPmid‐yearandannualreportstomakereportingeasierforCMPcoordinatorsandenhancetheusefulnessofdatacollected.Evaluatorsusedbothquantitativeandqualitativeapproachestoanalyzedatafromthesereports.
RevisionandadministrationofCMPcollaborativeeffectivenesssurvey.On‐linesurveysassessedIOGmembers’perceptionsofthequalityandeffectivenessoftheirlocalcollaborativepractices.Selectscalesfromthefirstyearadministrationwerere‐administeredinthe2010‐2011fiscalyeartoexplorechangesincollaborativeeffectivenessovertime.OMNIprovidedlocalresultstoindividualCMPsandanalyzeddataaggregatedacrossCMPstoassesscollaborativeeffectivenessstatewide.
Administrationoffamilyadvocacysurveys.IncollaborationwiththeFamilyVoiceandChoiceCommitteeandafamilyengagementresearcherfromWaldenUniversity,OMNIadministeredtwosurveysinSpring2011.OMNIcompiledthesedata,conductedaggregateanalyses,andprovidedareporttotheFamilyVoiceandChoiceCommitteetoinformfutureefforts.
Refinementsandtechnicalsupportoftheweb‐basedCMPportal.Establishedinthe2009‐2010fiscalyear,theCMPPortalisaweb‐basedsystemdesignedtofacilitatecommunicationandinformationsharingwithintheinitiative,providingaccesstocommondocuments,calendars,andresourcesinacentralizedlocation.BasedonfeedbackreceivedfromCMPstakeholders,evaluatorsmadefurtherrevisionstothecontentandstructureoftheCMPportalinthe2010‐2011fiscalyear.
EvaluationmethodstodatehavereliedoncollectingandanalyzinginformationaboutCMPstakeholders’perceptionsandotherself‐reporteddataduetotheabsenceofuniformprocessandmeasurementstrategiesacrossCMPs.Thus,findingsmustbeinterpretedwith
8
cautionastheymaybeinfluencedbystakeholderbiases.Asdiscussedabove,implementationofdatastandardsinyeartwowillsupportanalysisbetweenandacrosssitesbeginninginyearthree.
iii. Yeartwofindings
InformationprovidedbelowdescribeslocalCMPeffortstotargetthelegislativegoalsofHB1451andsummarizesfindingsfromthesecondyearofthestatewideevaluation.Findingspresentedinthisreportarenotexhaustive;additionalreportsanddeliverableshavebeenproducedduringthecourseoftheyear.Thelegislativegoalsprovidethestructureforthepresentationoffindingsthroughoutthisreport,witheachgoaldiscussedinturn.Ingeneral,eachreportsectionfirstdescribesCMPeffortstoaddresstherespectivegoal;exploresevidenceabouttheperceivedeffectivenessofthoseeffortstodate;andconcludeswithon‐goingchallengesorbarriersencounteredandopportunitiesforcontinuedimprovement.Thereportconcludeswithadiscussionofnextstepsanddirectionsforevaluationeffortsinyearthree.
9
LegislativeGoalI:Developamoreuniformsystemofcollaborativemanagementthatincludestheinput,expertise,andactiveparticipationofparentadvocacyorfamilyadvocacyorganizations.ThefirstlegislativegoaloftheCollaborativeManagementProgramsetsforthseveralexpectationsregardinglocalimplementationofacollaborativemanagementsystem.ItmandatesthatthemultipleagenciesaddressingtheneedsofColoradofamilies,youthandchildrendevelopamoreuniform,interagencysystemofservicemanagement.Itfurtherencouragescollaborativemanagementsystemstoengageparentsorfamilyadvocacygroupstoensurethesystem’sresponsivenesstothefamiliesitserves.ThefollowingsectionexaminesthedegreetowhichlocalCMPeffortshavecontributedtocreatinggreateruniformityinthecountysystemsaddressingtheneedsoffamilies,youthandchildren;moreeffectiveworkingrelationshipsbetweenagencies;andstrongerinvolvementofindividualsandorganizationsthatadvocateonbehalfoffamiliesservedbytheseagencies.WefirstdescribethemandatedcomponentsoflocalCMPsandthenexploretheextentofvariationamongprojectsinkeyareas.ThisdiscussionisfollowedbyapresentationoftheevidenceofoverallCMPeffectivenessandidentifiedchallengesorbarriers.Finally,weconcludethissectionwithadiscussionofCMPeffortstoinvolvefamiliesintheirwork.DataandinformationpresentedthroughoutthissectionwerecollectedfromCMPannualreportsandsurveydataoncollaborativeeffectivenessfromthe2010‐2011fiscalyear.
A. MandatedcomponentsofCMPs
Perthelegislation,countydepartmentsofhuman/socialservices,healthdepartments,judicialdistricts(probation),schooldistricts,andcommunitymentalhealthandbehavioralhealthorganizationsaremandatedtoparticipateinlocalCMPefforts.CMPsalsoarestronglyencouragedtoinvolvefamilyadvocacygroupsandothercommunityagencieswithrelatedmissionsandservicepopulations.Thelegislationalsostipulatesthattwoorganizationalstructuresaretobecreatedtosupportcollaborativeefforts:anInteragencyOversightGroup(IOG)tofacilitatecommunicationandresourcesharingacrossagencies;andanIndividualizedServiceandSupportTeam(ISST)toplanandcoordinatethedeliveryofservicesformulti‐agencyinvolvedfamilies.Additionally,CMPsarerequiredtoestablishprocessesthatfacilitateeffectivepartnerships,includingmethodsformakingcollectivedecisionsandimplementingproceduralchangeacrossagencies.
10
Finally,countiesmustsubmitMemorandaofUnderstanding(MOU)withparticipatingpartnersaddressingthefollowingareas:
Definitionofthepopulationtobeserved
Servicesandfundingsources CreationofanInteragencyOversightGroup(IOG) Developmentofcollaborativemanagementprocesses
DevelopmentofIndividualizedServiceandSupportTeams(ISST) Clearauthorizationtocontributeresourcesandfunding
Descriptionoftheprocesstoreinvestmoniessaved Performancebasedmeasures Confidentialityprotectionsandrequirements.
B. Localadaptation
Aswouldbeexpected,CMPsshareincommonmanycharacteristics,includingIOGandISSTcollaborativestructures,anemphasisonresourceallocationandreinvestment,andsimilardefinitionsoflocalperformancemeasures.However,CMPsalsoreflectaconsiderableamountofvariationinkeyareasincluding:
PatternsofIOGmembership,attendance,andareasoffocus; IndividualizedServiceandSupportTeam(ISST)modelsandprocesses;and CMPcoordinatorrolesandresponsibilities.
Eachoftheseareasisdescribedfurtherbelow.
i. IOGmembership,attendance,andareasoffocus
InformationcollectedinthefirstyearoftheevaluationrevealedthatthecompositionandactivitiesofIOGsvariedconsiderablyamongCMPs.YeartwodatacollectionalsoaddressedIOGattributes,includingIOGmembership,attendance,meetingfrequency,andareasoffocusduringthe2010‐2011fiscalyear.Analysisofthesedatashowedthatingeneral,bothmandatedandnon‐mandatedpartnersareengagedincollaborativeeffortsandthatthereexistmanycommonareasoffocusamongIOGs.
a) IOGmembershipandattendance
Inyearoneoftheevaluation,CMPsfrequentlycitedchallengestoobtainingbuy‐inandparticipationfromvariouskeystakeholders,mostnotably,families/parents,schools,andlawenforcementofficials.AsshowninTable2,however,membershipdataforthecurrentevaluationyearreflectrelativelysubstantialIOGparticipationofmanyofthesesamestakeholders,particularlylocalschools.Ingeneral,legislativelymandatedpartnerswere
11
Inclusionofnon‐mandatedpartnersonIOG
CMPsoftendescribedthepresenceofnon‐mandatedpartnersontheirIOGs: Nearlyall(89%)CMPsincludeSenateBill94staffontheirIOGs.
FourCMPsspecificallycitedtheinclusionofnon‐mandatedIOGpartnersasoneoftheirprimarysuccesses.Suchpartnersincludelocallawenforcementofficials,highereducationstaff,localtriballeaders,andothercommunitypartners.
wellrepresentedonIOGs,bothintermsofIOGmembershipandfrequencyofmeetingattendance.Infact,all(28)CMPsreportedIOGmembersfromcountydepartmentsofhumanorsocialservices,schools,andprobation,withthevastmajority(96%)attendingatleasthalfoftheIOGmeetingsintheyear.Ofnote,SenateBill94stakeholdersareincludedin25(89%)IOGsthoughtheirparticipationisnotrequired.
Table2.IOGmemberagenciesandattendance
AgencyorIndividualsRepresentedthroughMembershiponIOG
(n=28)
CountyDepartmentofHumanandSocialServices 28 (100%) 27 (96%)
SchoolRepresentative 28 (100%) 27 (96%)
Probation 28 (100%) 27 (96%)
Mentalhealthserviceprovidercenter 27 (96%) 25 (93%)
Substanceabuseserviceprovider 27 (96%) 24 (89%)
DivisionofYouthCorrections 27 (96%) 23 (85%)
Domesticviolenceserviceprovider 27 (96%) 23 (85%)
CountyHealthDepartment 26 (93%) 22 (85%)
SenateBill94representative 25 (89%) 22 (88%)
Behavioralhealthorganizationrepresentative 24 (86%) 20 (83%)
Localcourts/judicial 20 (71%) 13 (65%)
Diversion 13 (46%) 12 (92%)
Familyadvocacyorganization* 12 (43%) 10 (83%)
Lawenforcement 12 (43%) 7 (58%)
Familyrepresentative* 9 (32%) 0 (0%)
Localhealthservicesprovider 6 (21%) 6 (100%)
Electedofficial 6 (21%) 0 (0%)
Youthrepresentative* 4 (14%) 4 (100%)
BusinessorChamberofCommerce 3 (11%) 2 (67%)
Other 17 (61%) 16 (94%)
NumberofCMPsReportingIOGMember(%)
NumberofCMPsReportingIOG
MemberAttendingatLeast
50%ofIOGMeetings(%)
Note:Legislativelymandatedpartnersareshowninboldtextabove.*Insomeinstances,CMPsprovidedinconsistentresponsestosimilaritemswithintheAnnualReport.Assuch,countsofmembersreflectedabovemaydifferfromsimilarfigurespresentedelsewhereinthisreport.
Additionally,overallIOGcompositionreflectsthelegislativeexpectationofmulti‐agencymembershipreasonablywell.Onaverage,IOGsincludemembersfromabout12ofthe19
12
systemslistedinTable2,withaminimumof9andamaximumof16systemsrepresentedperIOG.Finally,nearlyall(27)IOGsmeetatleastquarterly,withabouthalf(15)meetingonamonthlybasis,andonlyonemeetinglessthanquarterly.
b) IOGareasoffocus
IOGsareexpectedtoconductoversightinseveralareas,includingimprovingcommunicationandresourcesharingacrosspartneragencies.Theyarealsograntedsignificantdiscretionindefiningtheirownpriorities.Inthe2010‐2011fiscalyear,IOGsidentifiedacommonsetofcoreactivitiesthatalignedcloselywiththelegislativegoals.ThefivemostcommonpriorityareasreportedarepresentedinTable3below,inorderoffrequency.Specificactivitiesassociatedwitheachpriorityareaalsoarelisted.
Table3.IOGpriorityareasandactivitiesinthe2010‐2011fiscalyear
PriorityArea ActivitiesIntroducingnewservicesEnsuringearlycontactbetweenprovidersandfamiliesRefiningISSTmodelsDevelopmentofIOGbylaws,clearpolicies,andproceduresdistributionofearnedincentivefundsFormationoftargetedIOGsub‐committeesClarificationofIOGmemberrolesandresponsibilitiesRecruitmentofnewpartners(lawenforcement,domesticviolence,tribalcommunities)CreatingopportunitiesforIOGpartnerstolearnmoreabouteachotherandtheCMPinitiativeHiringnewstaff(coordinators,facilitators)ProvidingtrainingandprofessionaldevelopmentopportunitiesStrategicplanningPursuedgrantfunding
AddressingCMPsustainability
Improvingqualityofservicedeliveryforchildrenandfamilies
Establishingand/orrefiningIOGstructure,processes,andprotocols
Strengtheningcollaborativerelationshipswithnewandexistingpartners
EnhancingCMPpersonnelresources
ii. IndividualizedServiceandSupportTeam(ISST)modelsandprocesses
TheISSTisaserviceplanningandcoordinatingbodythatbringstogetherprovidersfrommultipleagenciesand,insomecases,familyparticipants,todevelopanintegratedserviceplan.ISSTmodelsofservicedeliveryvariedsignificantlyacrossCMPs.YeartwooftheevaluationexploredISSTmodelsandprocessesbyexaminingthenumberofISST’simplementedperCMP,ISSTreferralsources,andagencyandfamilyinvolvementduringISSTs1.Findingsineachoftheseareasarepresentedbelow.
1SomeCMPsreportedmorethanoneISSTmodelorstructure.Thesecollaborativeswereaskedtofocusonone“primary”ISSTforthepurposesofrespondingtosubsequentquestionsaboutISSTcharacteristics.
13
a) NumberofISSTsandreferralstoISSTs
Onaverage,sitesreportedhavingtwoISSTs,withsixteenreportinghavingoneandonesitereportingutilizingsixseparatemodels.ThevariationinnumberandtheexistenceofmultipleISSTswithinasingleCMPreflectstheISSTmodel(s)beingused.Specifically,ISSTstypicallytakeoneofthreeforms:
ageneralISSTthatservesallfamilies;
aspecializedISSTwithinonebroadservicearena(e.g.,JuvenileJustice)thatservesonlyfamiliesreceivingrelatedservices;or
anISSTspecifictoanoutcomearea(e.g.truancyISST)thatservesonlyfamiliestargetingthespecificoutcome.
CMPswithmultipleISSTsusuallyhaveatleastonethatfocusesonaspecificissueorpopulation.
Table4.ISSTreferralsources
ReferralSource(Nvariesbyreferralsource)DepartmentofHumanServices 27 (96%) 40% (1% ‐80%)Schooldistrict 26 (93%) 24% (1% ‐66%)Probation 22 (79%) 15% (2% ‐45%)Mental/behavioralhealth 19 (68%) 9% (2% ‐25%)Court/judicial 13 (46%) 14% (4% ‐50%)Lawenforcement 10 (36%) 2% (1% ‐4%)DepartmentofPublicHealth 9 (32%) 6% (1% ‐22%)Self/parent 9 (32%) 11% (3% ‐25%)Diversion 7 (25%) 10% (5% ‐17%)DivisionofYouthCorrections 7 (25%) 5% (1% ‐8%)Other 14 (50%) 19% (3% ‐48%)
NumberofCMPs
Reporting(%)
Average%ofReferrals(Min.‐Max.)
YouthandfamiliesaretypicallyreferredtothelocalISSTthroughaCMPpartneragencyorotherentityinthecommunity.Table4(above)presentsthemostcommonreferralsources,includingthenumberofCMPsreportingeach,aswellastheaverageshareofreferralsmadebyeachsource.Thetwomostcommonreferralsources,bothofwhichalsomadethelargestshareofreferrals,arecountydepartmentsofhumanservicesandschooldistricts,respectively.Additionally,fourteen(14)countiesindicatedthattheyreceivereferralsfromsourcesnotlistedabove.Theseincludedomesticviolenceproviders,primarycarephysicians(pediatricians),SenateBill94staff,andearlychildhoodeducationproviders.
14
CMPimpactonstaffingteams
“We’vealwayshadgoodcollaboration,butithasgonetoadifferentlevelsincebecomingapartoftheCMP.Wearemoreeffectiveinourstaffing.Weusedtooverwhelmfamilieswithtoomanypeopleintheroom.ThankstoISSTandwrap,wehaverefinedourprocess.Wehavetherightpeopleintheroomattherighttime.We’vebroughtinmorefundingopportunitiesandprogrammingasaresultofourcollaboration.Andlastly,theroleofthefamilyintheirowncaseplanningisclearerandmoreeffective.”
b) ISSTstructureandagencyparticipation
BecauseISSTsaredesignedtoconvenerepresentativesfrommultipleagencies,CMPswereaskedaboutthenumberofagenciesparticipatinginatypicalISSTstaffing.Twenty‐six(26)indicatedthatatleastfouragenciesweretypicallyinattendance,withthirteen(13)indicatingparticipationofsevenormoreagencies.Themajority(59%)ofCMPsreportedthatISSTparticipationisbothstandardandtailoredtotheneedsoftheindividualyouthbeingserved;thatis,theyincludeacoresetofcommonparticipantsaswellasadditionalattendeesbasedonthespecificissuespresented.Justoverone‐quarter(26%)reportedthatISSTparticipationisdeterminedsolelyontheneedsoftheindividualyouth.CMPsalsodescribedthetypicalstructureofanISSTmeeting,namelyhowmanyfamiliesareaddressedduringthecourseofasinglemeeting,howmuchtimeisspentperfamily,andhowmanymeetingsareheldperfamily.Findingsareasfollows:
Projectsarenearlyevenlydividedbetweenthoseaddressingonefamilypermeetingversusthoseaddressingmultiplefamiliespermeeting(42%and46%,respectively).
Onaverage,CMPsdedicateaboutonehourperfamilyperISST,thoughthisvariedfromaboutfifteenminutestotwohours.
Onaverage,CMPsholdfiveISSTmeetingsperfamily,thoughthisvariedwidely,fromaminimumofonetoamaximumofthirty.2
iii. CMPcoordinatorrolesandresponsibilities
Inyearoneoftheevaluation,CMPsemphasizedthevalueofhavingadesignatedCMPcoordinator,ideallydedicatedtoCMPactivitiesonafull‐timebasis.Inyeartwo,CMPswereaskedtoprovideadditionalinformationabouttheircoordinators,particularlyaboutthecharacteristicsandrolesassociatedwiththisposition.Nearlyall(96%)ofCMPsreportedhavingadesignatedcoordinator,withthemajority(63%)servinginthisroleonafull‐timebasis.AbouthalfofCMP coordinatorsarehostedbythecountyDepartmentofHumanorSocialServices.Lesscommonhostagenciesincludementalhealthcenters,localfamily/youthadvocacyorganizations,thejuvenilejusticesystem,andlocalschools.Ofnote,
2WhenreportingtheaverageamountoftimespentinanISSTperfamilyandtheaveragenumberofISSTmeetingsperfamily,manyCMPsprovidedranges(e.g.,between1and3hours;orbetween10and15meetings).Intheseinstances,themid‐pointoftheestimatedrangewasusedforpurposesofanalysis.
15
aboutone‐thirdofCMPsreportedajointcoordinatorhost,oftenwithoneagency“housing”thecoordinatorandanotherservingasfiscalagentfortheposition. ThecoordinatorpositionisfundedfullywithCMPincentivefunds(i.e.,theyearlyfundsallocatedtoeachCMPbyCDHS)in14programs.However,six(6)projects(22%)reportedthattheircoordinatorpositionisfundedonlypartiallybyCMPincentivefunds,andseven(7)counties(26%)reportedthatthepositionissupportedentirelybyotherfunds.
Table5.CMPcoordinatorroles
CoordinatorRole(N=27)
Liaisonbetweenpartners 27 (100%)Disseminatescommunication 27 (100%)Documentsdecisionmaking 27 (100%)ISSTteammember 20 (74%)IOGleadership 19 (70%)Managesfundsandoverseesbudgets 18 (67%)CoordinatesISST 17 (63%)Managesintakeandreferrals 15 (56%)Conductsfollowupwithfamilies 13 (48%)Providesdirectservice 10 (37%)Other 12 (44%)
NumberofCMPsReporting
"Yes"(%)
CMPswerealsoaskedtoidentifythekeyrolesandresponsibilitiesofthecoordinatorposition(seeTable5,above,forthemostcommonresponses).CoordinatorsmostcommonlyprovidesupportattheIOG‐level,withall(100%)servingasliaisonsamongCMPpartners,disseminatinginformation,anddocumentingdecision‐makingprocessesandoutcomes.However,manycoordinatorsalsosupportserviceplanninganddelivery,includingparticipatingonandcoordinatingISSTs,andmanagingintake,referrals,andfollow‐upwithfamiliesserved.Twelve(12;44%)CMPsindicatedthattheircoordinatorhasresponsibilitiesotherthanthoselistedinthetable,suchasservingasaliaisonwithpeopleandagenciesoutsideofthelocalCMP(e.g.,CMPStateSteeringCommitteeandsub‐committees),managingdatacollectionandreporting,andsupportingfundraisingefforts.
C. EffectivenessoftheinteragencycollaborationestablishedbyCMPs
EvaluationeffortsalsoexaminedtheoutcomesofCMPcollaborativeprocessesandstructures.Inthefirstyearoftheevaluation,OMNIworkedwithcollaborationresearchersattheUniversityofDenvertoemploytwomethodstogatherinformationaboutcollaborativeeffectiveness.First,theyconductedkeyinformantinterviewswithasampleofIOGcoordinatorsandstate‐levelstakeholders.InterviewsexaminedprocessesusedtoestablishandimplementtheCMP,reflectionsoninclusionanddecisionmaking,and
16
lessonslearned.Thisworkculminatedinareportdescribingkeypracticesandperceivedindicatorsofcollaborativeeffectiveness(seeAppendixBforalinktothefullreport).Second,togatherabroaderperspectiveoncollaborativeeffectivenessandfurtherexplorethemesfromstakeholderinterviews,evaluatorsdevelopedasurveyforIOGmembers.TheCollaborationSurveyandOverallSuccessSurveywereadministeredto235IOGmembersin22CMPsinthe2009‐2010fiscalyear,andarevisedversionintegratingselectscalesfromthetwosurveyswascompletedby276IOGmembersin27CMPsinthe2010‐2011fiscalyear(seeAppendixBforalinktoreportsdescribingthesurveysinfurtherdetail).Thesurveymeasuresseveraldimensionsofcollaborativeeffectivenessthat,overall,reflecta)perceivedqualityofthecollaborativeprocess,andb)perceivedimpactofCMPsinreachingbothlegislativeandothergoals.AlthoughthemeasurereliesuponIOGmembers’perceptionsofeffectiveness,surveyitemsassessingthecollaborativeprocesshavebeenfoundtobepredictiveofpositiveoutcomesthatpublichealthcollaborativesaredesignedtoachieve.v
i. Qualityofcollaborativeprocesses
Overall,IOGmembersperceivetheircollaborativestohavestrongandproductivestructuresandprocessesinplace.Thesurveyincludedthreekeyscalesassessingthequalityofthecollaborativeprocess:
Structuralintegrity,whichoccurswhentheprocessisperceivedas“fair.”The
processallowsforsufficientopportunityforstakeholderstochallengeandrevisedecisions,butinacontextinwhichallpartnersfeelequallyheardandrespected
Authenticprocess,whichresultswhencollaborativepartnersperceivethedecision‐makingprocessas“openandcredible,”becausetheyseethemselvesashavingrealpowertobothformulateandmakebindingdecisionsvi
Strongleadership,whichreflectstheperceptionthateffortsofcollaborativepartnersareledbydedicatedandeffectivecoordinators.
Onaverage,IOGmembersindicatedthattheirCMP’seffortswerecharacterizedbyhighqualityexperiencesineachofthesethreeareas,withaveragescoresexceedingthe4.25markerofcollaborativeeffectivenessona6‐pointscale,withhigherratingsindicatinggreatercollaborationvii.Ingeneral,therewaslittlevariabilityintheaveragescoresintheseareasacrossCMPs.
17
Figure2.Ratingsofthequalityofcollaborativeprocess
Note:Itemsareratedona6pointscale(1=stronglydisagree,2=disagree,3=disagreemorethanagree,4=agreemorethandisagree,5=agree,6=stronglyagree).N=276respondentsfrom27IOGs.
ii. Successofcollaborativeprocesses
IOGmembersalsoratedtheextenttowhichtheircollaborativeprocessesledtosystems‐levelsuccessesduringthe2010‐2011fiscalyear.AsseeninFigure3,onaverage,IOGmembersperceiveahighlevelofgeneralsuccess,believingthattheirCMPsgenerallyhaveaccomplishedorexceededtheiryearlygoals.Additionalevidenceofcollaborativeeffectivenesswasgatheredinannualreports,inwhichCMPsdescribedsignificantsuccessesrealizedasaresultoftheircollaborativeeffortsduringthe2010‐2011fiscalyear.Themostcommonlymentionedareasofsuccess(mentionedbymorethanfiveCMPs)appearinTable6,alongwithexamplesofreportedachievementsineacharea.
18
Figure3.Item‐levelratingsofoverallsuccessofCMPduringtheFY2010‐2011
1 2 3 4 5 6
Hasaccomplisheditsspecificgoals
Hasachievedmorethanitsoriginalgoals
Hasledtonewprojectsorefforts
Hasachievedextraordinarysuccess
Note:Itemsareratedona6pointscale(1=stronglydisagree,2=disagree,3=disagreemorethanagree,4=agreemorethandisagree,5=agree,6=stronglyagree).N=276respondentsfrom27IOGs.
Table6.CMP‐reportedevidenceofcollaborativeeffectiveness
AreasofCollaborativeEffectiveness ExampleAchievements
Establishmentofsubcommittees,leadingtomoreefficientuseofIOGmeetings•asteeringcommitteetooversee"day‐to‐day"operationalactivities•anoutcomescommitteetospecifylocalevaluationmeasuresandgoalsFullpartnerparticipationinstrategicplanningsessionsCreationofyouthadvisoryboardEstablishmentofa"virtualvoting"processwithinaruralIOGtoincreaseactiveparticipationindecision‐makingRoll‐outofnewprogramstargetingareassuchaspregnancy,mentoring,youthservicelearningIncreasesinthenumberoffamiliesreceivingspecificservicesIncreasesinthenumberofreferrals,indicatinggreatercommunityawarenessofeffectivenessofservicedelivery
DecreasedtimebetweenreferralandenrollmentininterventionservicesFundedsuccessfulgrantwritingeffortsBlendedorbraidedfundingidentifiedforspecificservicesleadingtomoreintegrationandstreamlinedservicesamongpartneragenciesEstablishmentofprocesstograntseedfundstonewprograms
ImprovedprocesseswithintheIOG
Developmentofnewprogramsor
enhancementofexistingservicedelivery
Supportingsustainability
19
PositivecommunityimpactsasaresultofCMPprocesses
CMPswereaskedtoreflecton“whathaschangedinyourcommunitysincetheinitiationofCMP.”“OneofthebiggestimpactstheCMPhashadin[ourcounty]isincreasedcommunicationbetweenagencies.Thepartneragenciesareworkingmoretocollaborate,shareresourcesandreduceduplicationofservices.Asasmallcommunity,relationshipsalreadyexistedandhaveonlybeenstrengthenedbytheirmembershipintheCMP…largeroverarchingissuesgetaddressed…andspecificplanstoaddressareasofconcerncanmoreeasilybecollaboratedupon...”“Thereisgreatercommunicationbetweentheagencies,namelytheschoolsandDSS.Inthepast,therehasbeensomecontentionbetweenthetwoaroundwhoisresponsibleforhelpingthisfamilyandwhowillpayforit.Forexample,ayouthwouldbetruant,andinsteadofutilizinganRTIprocess,theschoolwouldpuntthekidtoDSSandaskthemtofileaD&N.Thisputtheagenciesatoddswitheachother.BecauseoftheCMP,weareseeingsystemsbeginningtorecognizewhatisnotworkingintheirprocessandwantingtoaddressit.Insteadofpointingfingers,we’rehavingmoreconversationswithawin/winattitude.Youtharenotbeinglabeledas“YourProblem”,ratherthecommunityasawholeisembracingtheirneedscollectively.Thisisatremendousstepintherightdirection.Trustisgrowingandsystemsrealizenooneisouttotearthemdownfortheirmistakes.RathertheIOGisheretohelpeachothersucceedforthebestinterestoftheyouth.”“[Our]county’sCollaborativeManagementProgramisembarkingonitsfifthyear.Eachyearhasshownincreasedcollaborationandeffectiveservicedeliveryamongyouth‐servingagenciesasthemodelofcollaborationsupportedbytheCMPgraduallyhasbecomemoreofaculturalnormwithinourcommunity.TheIOGanticipatesthattheCMPwillcontinuetofine‐tuneitseffortsandthatitssuccesswillcontinuetogrow.”“Thoughagencieshavealwaysworkedtogether,thisefforthasprovidedtheresources(coordinator,fundingforsomeservices)thatallowtheagenciestostepbackandlookintosystemsandknowthatthereisamechanismthatwillhelpsolveissuesthatarise.1451hasbecomeahouseholdwordamongouragencies–peopleknowwhatwearetalkingaboutwhenwesay1451.Agenciesknowthat1451isagoodcollaborativepartnerinmeetingtheneedsofhighriskchildrenandfamilies.”“OurcountyhasexperiencedasenseofunityandpurposethatisdirectlytiedtotheCMP.Weareidentifyingweaknessesandfeelingthateffectivestrategieswillbefoundtosolvelocalproblems.”
20
iii. ChallengesassociatedwithCMPcollaborativestructuresandprocesses
Takentogether,theresultsaboverevealedthatIOGmembersgenerallybelievethattheyhaveestablishedhigh‐qualityandeffectivecollaborativepartnerships.Despitetheseaccomplishments,CMPsstillstrugglewithsomecommonissues.Themostcommonchallenges,alongwithstrategiesusedtoovercomethem(ifidentified),include: Participationandbuy‐infrompartners.Eight(8)CMPsciteddifficultiesobtainingbuy‐in
frompartners,bothinCMPoversightandservicedelivery.SomeoftheseCMPsindicatedthatengagingspecificpartners,suchasschools,non‐profits,andcitygovernmentagencies,hasbeendifficult.OthersdescribedsituationsinwhichIOGmembersagreeuponaplan,butthenmayfailtocommunicatethistothespecificstaffmembers(s)responsibleforimplementation.CMPsindicatedthattheyareworkingtoaddresstheseissuesbyincorporatingexpectationsaboutparticipationandinternalcommunicationintoagencypoliciesandcounty‐levelMOUs.
CMPstaffing.Anequalnumber(8)ofCMPsreportedthatinadequatestaffingandturnoverhaveimpactedCMPadministrationandservicedelivery,citingdifficultiesresultingfromthelackofacoordinator,ISSTfacilitator,orotherstaff.SomeCMPshavecontractedwithprivateprovidersoragenciestoaddressstaffingneeds.
Financialstability/sustainability.Seven(7)CMPsspecificallyreportedchallengesassociatedwithageneralshortageoffunding,largelyattributedtotheeconomicdownturn.Inordertoaddressthesechallenges,CMPsindicatedthattheyareundertakingadditionaleffortstoobtainexternalfunding.
Informationsharing/confidentiality.Barriersrelatedtoclient‐levelinformationsharingwerereportedbysix(6)CMPs.Inmostinstances,CMPsdescribedalackofpartneragreementregardingconfidentialityrequirements.StrategiesintendedtoclarifytheserequirementsincludeconsultationwiththeColoradoChildandYouthInformationSharingCollaborative(CCYIS),aswellasgeneraldiscussionamongpartners.
UnderstandingandclarityofCMPmission.Finally,three(3)CMPsciteddifficultyensuringclearandcommonunderstandingofcollaborativegoals,activities,andpracticesamongpartners.OnefactorcontributingtothischallengeappearstobeturnoverattheIOG‐level,aswellasatpartneragencies.Toaddressthischallenge,someCMPshaveconductedresearchoncollaborativeeffectiveness,consultedwithCMPcoordinatorsinothercommunities,andheldregularmeetingstodiscusstheseissues.
Overall,CMPsdescribedtheirlocalcollaborativestructuresandrelationshipsasstrong,andreportedthatjointeffortshaveledtosignificantimprovementsinservicedeliverysystemsandcapacitytoaddresscommunityneeds.AlthoughCMPscontinuetoexperiencesomebarriers,theyappeartobeworkingtogethertoidentifysolutionsandinnovationstoovercomethese.
21
D. Inclusionoftheinput,expertise,andactiveparticipationofparentorfamilyadvocacyorganizations
TheestablishinglegislationstronglyencouragesCMPstoengageparentorfamilyadvocacygroupstoensurethatsystemsareresponsivetothefamiliestheyserve.AlthoughthelegislativelanguagefocusesoninvolvementoffamilyrepresentativesattheIOGlevel,theCMPStateSteeringCommitteehasencouragedCMPstoalsoincorporatefamilyadvocatesandfamilymembersattheISSTlevel.Astatefamily‐drivenorganization,theFederationofFamiliesforChildren’sMentalHealth,hasbeenastrongproponentandfacilitatoroftheseefforts.Familyrepresentativescontributeuniqueperspectivesnotonlytopolicybutalsotothecontentanddeliveryoffamilyservices.Forexample,familyrepresentativesmayidentifypotentialchallengesforfamiliesinserviceplansorpolicies,whichcaninformeffortstodevelopmoreintegratedandstreamlinedprocessesthatresultinbetterfamilyoutcomes.ThissectionpresentsfindingsrelatingtofamilyinvolvementinCMPactivities,includingstrategies,evidenceofeffectiveness,andon‐goingbarriers.Weincludeinformationontheextenttowhichfamilyandyouth,aswellasfamilyadvocates,areincludedinCMPactivitiesanddescribeevidenceoftheeffectivenessoffamilyengagementstrategiesandcommonbarriersencountered.Thisisreportedatthreelevels:
InvolvementoffamilyrepresentativesinIOGefforts; InvolvementoffamilyrepresentativesadvocatingforfamiliesinISSTs;and Engagementoffamiliesintheirownservicedeliverysystemsandprocesses.
i. InvolvementoffamilyrepresentativesinIOGefforts
AttheIOGlevel,familyinvolvementismostreadilyreflectedbythepresenceoffamilymembers,youth,and/orfamilyrepresentativesontheIOGitself.Table7belowprovidesthenumberandproportionofCMPsreportingIOGmembersineachofthesethreecategories.Over40%ofCMPsreportedatleastonefamilymemberontheirIOG,about40%reportedatleastonefamilyadvocate,andabout20%reportedatleastoneyouthmember.
22
Table7.FamilyrepresentationonIOGs
0 17 (59%) 23 (79%) 18 (62%)1 8 (28%) 3 (10%) 5 (17%)2 4 (14%) 2 (7%) 4 (14%)3 0 (0%) 0 (0%) 1 (3%)4 0 (0%) 0 (0%) 1 (3%)5+ 0 (0%) 1 (3%) 0 (0%)
NumberofRepresentatives
onIOG
FamilyMember(%)
(N=29)
YouthMember(%)
(N=29)
FamilyAdvocate(%)
(N=29)
NumberofCMPsReporting
TheextentoffamilyinvolvementintheIOGisreflectednotonlybythepresenceoftheserepresentativesbutalsointheirroleasadecision‐maker.Onlythree(3)oftheCMPsthatdohavefamilyrepresentativesontheirIOGsincludethemasnon‐votingmembers.
ii. InvolvementoffamilyrepresentativesadvocatingforfamiliesinISSTs
Familyinvolvementalsooccurswithinserviceplanningeffortsonbehalfofotherfamilies.NearlyhalfofCMPs(12CMPs,46%)reportedthatfamilymembersarealwaysviewedaspartnersinserviceplanningforotherfamilies,andthevastmajority(85%)ofCMPsreportedthatfamilymembersareseeninthiswayatleast“sometimes.”WhileCMPsreportedasignificantdegreeoffamilymemberinvolvementforthebenefitofotherCMPfamilies,few(22%)indicatedthattheyhaveaprocessinplacetosupportormentorfamilymembersservinginthisrole.Thesix(6)CMPsthatdidreporthavingsuchaprocessinplacedescribedthefollowingtypesofsupport:
orientationprocessestoclarifyCMPmissionandactivitiesaswellastheroleoffamilypartners;
generalsupervision,coachingormentoringprovidedbyotherCMPpersonnel;and assistanceobtainingexternaltraining.
Localhighlight:IOG‐levelyouthadvisoryboards
SomeCMPsdescribedinnovativestrategiesforincludingyouthinCMPoversightactivities.Forexample,oneCMPreportedtheimplementationofaYouthAdvisoryBoardthatworksincollaborationwithitsIOG.TheBoard,whichiscomprisedof10localyouth,wasestablishedduringthecourseofthe2010‐2011fiscalyearandmeetsindependentlyoftheIOG,providesinputonIOGactivities,andvotesonallofficialIOGissues.
23
Lessthanhalf(42%)ofCMPsemploypaidorvolunteerfamilyadvocates(alsoreferredtoasfamilysystemsnavigatorsandfamilysupportpartners)toworkwithfamiliesgoingthroughtheISSTprocess.AdditionaldatawerecollectedfromCMPsthisyeartobetterunderstandtheroles,training,background,andresourceneedsoffamilyadvocates;findingsaresummarizedinthehighlightboxbelow.
FamilyInvolvementSurvey:RoleoffamilyadvocatesinserviceplanninganddeliverySince2007‐2008,theFamilyVoiceandChoiceCommittee(FV&C)hasbeencollectingdataaboutCMPfamilyinvolvement.Previousyears’datacollectioncenteredonfamilyinvolvementattheIOG‐levelwhereasthisyear’seffortsfocusedonlearningaboutfamilyadvocacyactivities.SurveysweredesignedinpartnershipwiththeFV&CCommittee,OMNI,andDr.DonnaHeretick,afamilyengagementresearcheratWaldenUniversity,andadministeredinthespringof2011.Twenty‐fourcoordinatorsand42individualsservinginafamilyadvocacyrolecompletedsurveys.Highlightedfindingsareasfollows: Of24totalcoordinatorsurveyparticipants,42%(13)reportedhavingfamilyadvocatesinvolvedinISSTs.
ThemajorityofCMPswithfamilyadvocatesreportedthattheyarepaid,mostoftenbythelocalDHS,andaboutone‐third(36%)devotemorethan20hoursperweektothiswork.
CMPfamilyadvocatesreportedavarietyofrolesandresponsibilities,themostcommonofwhichwereparticipatinginIOGmeetings;consultingwithfamiliesabouton‐goingissues;referringfamiliestolocalservicesandsupports;andassistinginfamily‐levelprogramorservicedesign.
Overhalf(55%)ofadvocatesreportedcaseloadsof10orless,andaboutone‐thirdreportedlargercaseloadsofanywherebetween11‐30families.
Familyadvocatetrainingmostfrequentlyinvolvesinformalandformalagency‐basedtraining,specializedworkshops;andinformation‐sharingamongpractitioners,bothon‐andoff‐the‐job.
Whenaskedtodescribethegreatestchallengesorresourceneedsforenhancingfamilyadvocacywork,themostcommonresponsesincluded:
o TrainingonprofessionalrolesandresponsibilitiesforFamilyAdvocates(e.g.,confidentialityprotocols,communicationpractices);
o SupportanddevelopmentofFamilyAdvocateswithpersonalexperiencewiththeCMP,namelypreviousCMPclients;
o Developmentofahiringmanualandformalpolicies/proceduresaboutFamilyAdvocatesforCMPs;and
o Additionalfundingforadvocatepositionsandassociatedcosts.
24
iii. Engagementoffamiliesintheirownservicedeliverysystemsandprocesses
AcorevalueoftheCMPinitiativeismeaningfulinvolvementoffamilymembersindevelopingtheirownserviceplans.Ingeneral,CMPsreportedasubstantialdegreeoffamilyengagementinexploredareasincludingthetimingoffirstcontactwiththefamily,ISSTparticipationbyaprimarycaregiver,andthefrequencywithwhichimportantdecisionsaremadewiththefamilypresent.Themajority(63%,17CMPs)ofCMPsindicatedinvolvingfamiliesimmediatelyafterbeingdeterminedeligibleforISSTservicesandnearlyone‐third(30%,8CMPs)reportedthatfamiliesarefirstcontactedpriortoadecisionregardingISSTeligibility.AsshowninTable8,nearlyallCMPs(96%)reportedthatprimarycaregiversparticipateintheISSTeitherfrequentlyoralways;themajority(77%)reportedthatimportantdecisionsaboutservicesarerarelyornevermadewithoutthefamilypresent;andthevastmajority(89%)reportedthatthefamilyfrequentlyoralwaysparticipatesintheprocessofdevelopingandwritingserviceplans,andisthengivenacopyofthatplan(93%).
Table8.ISST‐levelfamilyinvolvementpractices
Always 19 (70%) 6 (22%) 1 (4%) 19 (70%) 21 (78%)Frequently 7 (26%) 6 (22%) 0 (0%) 5 (19%) 4 (15%)Sometimes 0 (0%) 11 (41%) 5 (19%) 2 (7%) 2 (7%)Rarely 1 (4%) 3 (11%) 11 (42%) 1 (4%) 0 (0%)Never 0 (0%) 1 (4%) 9 (35%) 0 (0%) 0 (0%)
ImportantDecisionsMadeWithoutChildorFamilyPresent
(n=26)
FamilyParticipatesinISSTServicePlanning(n=27)
FamilyGivenaCopyofISSTPlan
(n=27)
NumberofCMPsReporting(%)
Frequency
AtLeastOnePrimaryCaregiverParticipateson
ISST(n=27)
Family,Friend,AdvocateOtherthanFamilyon
ISST(n=27)
iv. EffectivenessoffamilyinvolvementinCMP
Toalargedegree,themeasureofeffectivenessofthislegislativegoalisreflectedbythenumbersofCMPswithfamilymembersinvolvedinIOGsandISSTs,aswellastheextenttowhichCMPsreportthatthesemembersarevaluedasactivepartnersinCMPservicedelivery.Thefindingspresentedabovesuggestthatprojectsaremakingstridesinthisarea,withapproximatelytwo‐thirdsofCMPsreportingactiveparticipationoffamilyrepresentativesintheirIOGsandalmostallCMPsindicatingactiveinvolvementoffamiliesinISSTmeetings.Additionally,someCMPstakeholdersspecificallyhighlightedachievementsaroundfamilyinvolvement,withonereportingthat“agenciesarebeginningtoletgoofcertainagendaswithyouthandfamiliesandprocessingwiththefamiliestodecide
25
whattheywanttoworkon”andanotherthat“systems[are]understandingfamiliesbetter,whichcreatesbetterplansandoutcomesbasedonthefamilies`strengthsandculture.Moreindividualizedgoalsandplanscreatebettersuccess.”Some,butnotallCMPs(59%),reportedthattheyhaveastructuredorformalon‐goingprocesstogatheradditionalevidenceofeffectivenessinfamilyinvolvement.TheseCMPsmostcommonlydescribedthecollectionoffamilysatisfactionsurveys(69%),followedbytrackingfamilyparticipationinISSTmeetings(63%),andtrackingfamilyrepresentativeparticipationinIOGmeetings(38%).Someofthelearningsfromtheseeffortsarehighlightedabove.Further,manyCMPsprovidedanecdotalevidenceoftheeffectivenessoftheireffortsinthisareaviastoriesofhowfamilyengagementleddirectlytoimprovementsinservicedeliveryandpositiveoutcomesforchildrenandfamilies.Someofthesestoriesaresharedonthefollowingpage.
Localevaluationoffamilyinvolvementefforts
Threesiteshighlightedinformationobtainedviaeffortstoevaluatetheirfamilyinvolvementefforts:
“Thevastmajorityoffamiliesthathavecompletedthesurveyfeelthattheirvoiceswereheard,theirconcernsandneedsconsidered,andtheirchildren’sneedsmetthroughtheCMP.”
“WehaverealizedthatwhenmorefamilymembersareinvolvedintheISSTprocess,theyaremoreengagedintheprocess,thetargetedyouthismoresupportedandconfidentinthemeeting,andtheyaremorelikelytocontinuetoengageinfollowup[ISST]conferences.”
“Family/youthpostservicesurveysindicatefamiliesfeeltheywereanequalpartoftheprocess,thattheprocesswasrespectful,andthattheirperspectiveswereprioritized.”
26
v. Challengestofamilyinvolvement
Whiletheinitiativehasseennotablegrowthinthedegreeoffamilyinvolvementoverall,CMPsalsoidentifiedanumberofbarriersbelievedtolimiteffectiveengagementoffamilies.AsillustratedinTable9below,themostfrequentlycitedbarrierwasidentifyingfamiliesappropriateforIOGparticipation(64%),followedbyalackoffamilyknowledgeorexperienceaboutCMPprocessesandconfidentialityconcerns(39%;seeTable9).
Table9.CMP‐reportedbarrierstofamilyinvolvement
IssuesidentifyingappropriatefamiliesforIOGparticipation 18 (64%)Confidentialityissues 11 (39%)Lackofknowledge/experienceoffamilyoryouthrecruitedaboutCMPprocesses 11 (39%)Timeconstraints/scheduling 7 (25%)Lackofcommitmentamongrecruitedyouthandfamilies 6 (21%)Lackoffundingtocompensatefamilyandyouthmembers 6 (21%)Geography/distance 3 (11%)Personnelturnover 1 (4%)Otherbarriers 7 (25%)
BarrierDescription(N=29)
NumberofCMPsReporting(%)
Note:Totalssumtogreaterthan29CMPsand100%becauseCMPscouldselectupto3barriers.
CMPsdescribedavarietyofapproachestoaddressthesebarriers.Selectstrategiesarelistedbelowandillustratethewiderangeofmethodsbeingusedtofosterfamilyinvolvementstatewide:
Caseexamplesofeffectivefamilyinvolvement“The[ISST]metwiththechildandfamilytodiscusswayswecouldsupportthechildtomaintainaneducationalsetting,inthedelinquencysystemandathome.Thefamilyrequesteddaytreatmentservices,coaching(mentoring)servicesandinhometherapy.[TheISST]fundedallthreeservicesandcontinuedtomeetwiththefamilyonamonthlybasistoaddressanyconcernsandhelpthefamilyaccomplishtheiridentifiedgoals.Thechildisnowonareducedschoolscheduleandisgettinghomeboundservicesthroughtheschooldistrict.ThefamilyandchildhavesuccessfullycompletedtheirinhomefamilytherapyprogramandtheyouthhasnotgottenanynewchargessinceJanuaryofthisyear.DHShasclosedtheircaseastherearenolongeranybeyondcontrolofparentissues.”OneCMP‐servedfamily“mentionedthattheyappreciatedthefactwelistenedtothemandhelpedthembrainstormideasversustellingthemwhattheyshoulddo.Using[theISST],theFacilitatorwasabletocoordinatewiththeschooltoletthemknowthatthedaughterwasonnewmedication,thefatherreceivedcounselingthroughourmentalhealthpartner,andthefamilyworkedthroughthe[ISST]process.Schoolreportsthedaughterisdoingmuchbetterinschool;anddadisstableonhismedicationandcontinueswithcounseling,andtheparentscontinuetoco‐parentthechildren.”
27
Recruitment/retention.SomeCMPsdescribedstrategiesfocusedontherecruitmentoffamilymembersandrepresentativeswithappropriateknowledgeofandexperiencewiththeCMP,includingoutreachtograduatesfromlocalprograms,aswellasprovidingfinancialcompensationtofamilyrepresentativesinrecognitionoftheircontributions.
Traininganddevelopment.CMPsalsodescribedeffortstosecureandprovidefinancialsupporttoenablefamilyrepresentativestoattendtrainingsdeemedrelevanttotheirrolesandresponsibilitieswithintheIOG.
Planning.Finally,someCMPsdescribedstrategiesintendedtoformalizetheirfocusonfamilyinvolvement,includingthecreationofaYouthAdvisoryCouncil,whichinformsIOGprocessesanddecision‐making,aswellasthedevelopmentofformalplanningofeffortstoincreasefamilyinvolvement.
Insummary,the31countiesparticipatingintheCMPinitiativehaveaddressedthislegislativegoaltopromotegreateruniformityinthesystemsservingColoradofamilies,children,andyouth,byimplementingspecifiedorganizationalstructures,suchasIOGsandISSTs,andprocessessuchasfamilyinvolvement,necessarytoimplementasystemofcollaborativemanagement.AlthoughlegislationhassetsomecommonstandardsforCMPs,therehasbeenconsiderableleewayforlocalsitestoadapttheirstructuresandprocesses.TheevaluationrevealedthatIOGsacrossthestatesharesimilarprioritizedgoals,butsetupdiversestructuresandactivitiestoachievegoals(e.g.,intermsofthemembershipandoperationsoftheirIOGs,ISSTs,andstaffing).Overall,sitesperceivetheirlocalcollaborativestructuresandrelationshipstobestrongandeffectivelyresultinginimprovementsintheirservicedeliverysystemsandcapacitytoaddresscommunityneeds.Inparticular,CMPsdescribedsignificantprogresstowardincreasedandmeaningfulfamilyinvolvementatalllevels.AlthoughCMPscontinuetoexperiencesomebarriers,theyreportedactiveeffortstocontinuetolearnfromandbuildupontheirsuccesses,aswellasspecificstrategiesintendedtoaddressbarriersencounteredtodate.
28
LegislativeGoalII:Reduceduplicationandeliminatefragmentationofservicesprovidedtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices.ThesecondlegislativegoaloftheCollaborativeManagementProgramistointegratemulti‐agencyservicesforfamiliesandchildren,suchthattheduplicationandfragmentationinservicedeliveryisreducedoreliminated.Serviceduplicationreferstosituationswherefamiliesreceivethesameservicesfrommultipleagencies.Fragmentationreflectstheconditionofuncoordinated,conflicting,orunrelatedserviceprovisionwhentheseservicesarenotmanagedorcoordinatedacrossagencies.Bothissuescanbeaddressed,atleastinpart,throughtheprovisionofinteragencyplanningandservicemanagementefforts.Sucheffortscanresultinthealignmentofrequirementsforfamilies,easingofmulti‐agencynavigation,andthecreationofcostefficienciesbyeliminatingunnecessaryduplicationofservices.CMPsreportimplementingavarietyofstrategiesdesignedtoimpactduplicationandfragmentationofservicedelivery.Theseinclude:a)coordinationofclientconsentsandassessments;b)integratedserviceplanning;c)arrangingpaymentforandprovisionofservices;andd)informationsharingacrossagencies.Activitiesineachoftheseareasareoutlinedbelow,followedbyadiscussionoftheirperceivedeffectiveness,andrelatedbarriers.DataaboutthestrategiesbywhichCMPshaveworkedtoaddressbothduplicationandfragmentationweredrawnfrommid‐yearandannualreportsandsurveydataoncollaborativeeffectiveness.
A. Coordinationofclientconsentsandassessment
Projectcoordinatorswereaskedwhethertheyusecommonprocessesanddatacollectionformstoobtainclientconsentsandassessmentinformation.Amajority(74%)ofCMPsuseacommonconsentformthatissharedacrossallagencies,meaningthattheindividualorfamilyneedsignonlyoneconsentform.Mostoften,thiscommonreleaseoccursattheISSTandenablesthesharingofclient‐levelinformationacrossallagenciesinvolvedinagivencase.However,three(3)CMPsindicatedthatnotallpartneragenciesareincludedassomeagenciesrequireadditionalconsentfromyouthorfamilies,andotherselectnottousethecommonform.Two(2)CMPsindicatedthat,whiletheydohaveacommonconsent,individualyouthorfamiliesareallowedtospecifywhichagenciesmayandmaynothaveaccesstotheirinformationortorevoketheirconsentatanytime.AbouthalfofCMPs(14)reportedhavingacommonclientassessmentformandprocessthatissharedacrossagencies.Whilethespecificformvaries,themostcommon
29
CMPimpactonserviceplanninganddelivery
OneCMPreportedthat“initiatingtheCMP…hasshiftedtheperspectiveofpartneringorganizationstothatofamorecollaborativefocusonprovidingthoseservices.AswedevelopandfullyimplementourISSTprocedures,therewillbeamoreevidentimpactontheactualdeliveryofdirectservices.”
assessmentsarethoseusedwithinthejuvenilejusticesystem[e.g.,ColoradoJuvenileRiskAssessment(CJRA),MassachusettsYouthScreeningInstrument(MAYSI)].Someoftheseassessmentsareextensive(e.g.,CJRA)andhavebeenwidelyadoptedorarerequiredbymultipleprogramsinColorado(e.g.,SB94,diversionprograms);thus,sharingassessmentdatareducesduplicationinproviderandfamilytimeandeffort.OneCMPdescribedtheexpectedrelationshipbetweenitscommonclientassessmentandeffortstoreduceduplicationandfragmentationinthefollowingway:[allclientsservedbytheISSTprocess]“engageinacomprehensiveassessmentprocess.Theinformationcollected…isdocumentedbytheISST’scoordinatorandsharedwithallagencieswhoparticipateontheISSTinordertohelpavoidduplicationofservicesandensurethecontinuityofprovidedservices.”
B. Integrationofservicedeliveryplanning
Anothercomponentofcoordinatedserviceplanninganddeliveryisthedevelopmentofanintegratedserviceplan.ThelegislationrequiresISSTstodevelopservicedeliveryplansforeachstaffedyouthand/orfamily.Whilethestructureoftheseplansisnotdictated,thelargergoalsoftheinitiative(i.e.,tofosterintegratedandappropriateservicedelivery,toreduceduplicationandeliminatefragmentationofservices)wouldsuggestthatasingle,individualizedplanbedeveloped.CMPswereaskedtodescribethedegreetowhichtheirISSTplanspossessthesetwokeycharacteristicsandreportedthefollowing:
AllserviceplansdevelopedduringISSTprocessesareindividualizedatsomelevel,withthemajority(78%)indicatingthatplansareindividualizedtothefamilybeingserved,and22%indicatingthatplansareindividualizedtotheyouthbeingserved.
Additionally,nearlyall(92.5%)ofCMPsreportedthatISSTserviceplansareintegrated.However,onlyhalfofthese(13)indicatedthatasingular,integratedserviceplanisdevelopedandsharedamongproviders.Somesitesdescribedthatpartneringagenciesagreeuponservicecomponentsyetcreatemultiple(i.e.,agency‐specific)plans.Whiletheseplansmayreflectsomeintegrationofservices,thepresenceofasingle,agreed‐uponplanmaysignaltofamiliesagreatercommitmentamongallagenciestotruecoordinationandsharedresponsibilityofservices.
C. CoordinationofpaymentforandprovisionofservicestoCMPyouth
CMPsdescribedstrategiesbywhichresponsibilityforcostsand/ordeliveryofservicestoyouthandfamiliesaresharedamongpartneragencies.Over80%ofCMPssharethecosts
30
ofservicedeliveryonacase‐by‐casebasis,withindividualagenciesdesignatedtopayforparticularcomponentsofeachindividualserviceplan.Veryfew(7%)CMPsreportedhavingashared,generalfund,fromwhichservicesarepaid.CMPsalsodescribedthefrequencywithwhichpartneragenciesareexpresslydesignatedtoeitherpayforand/ordeliverservicestotheyouthorfamiliestheyserve.Nearlyallsitesindicateddesignatingaspecificagencywitheachoftheseresponsibilitieseitheroftenoralways(seeTable10below).Processessuchastheseservetoclarifyagencies’rolesinservicedeliveryandmaybeviewedasindicatorsofcollaborationandmoreefficientservicedelivery.
Table10.CMPdesignationofagencyserviceresponsibilities
Always 10 (37%) 19 (70%)Often 15 (56%) 7 (26%)Sometimes 1 (4%) 1 (4%)Rarely 1 (4%) 0 (0%)Never 0 (0%) 0 (0%)
NumberofCMPs(%)(N=27)
Frequency Agency(ies)aredesignatedtopay
forservices
Agency(ies)aredesignatedto
provideservices
CMPsalsotargetconflictingmandatesortreatmentrequirementsacrosspartneragenciesinordertoreduceduplicationandfragmentationintheirlocalsystems.All29CMPsreportedsomereductioninthisareaandoverhalfoftheCMPs(15)indicatedreducingduplicationandfragmentation“alot,”througheffortssuchasclarifyingandsimplifyingpoliciesandreducingconflictingrequirements.AnexamplereflectedinaCMPMOUstatesthat“Whenanexisting[ISST]caseexistsandplacementdecisionisrequired,thediscussionanddecisionwilloccurwithinthe[ISST]meetingandwillnotalsogothroughthe[establishedlocalDHSplacementteam]process."Thistypeofagreementstreamlinesprocessesanddecision‐makingtobetterservefamiliesinthecounty.
D. Informationsharingpractices
Thecreationofsystemsthatfacilitatethesharingofclientdataandrelatedinformationacrossagenciesrepresentsanotherstrategythatsupportsinter‐agencycoordinationandserviceintegration.Allowingstafffrommultiplesystemstoaccessacommonsetofclient‐leveldatacreatesopportunitiestoidentifyandaddressinstancesofserviceduplicationand/orfragmentation.SomeCMPsreportedhavingestablishedsuchsystemsbothforthe
31
purposesofsharinginformationgenerally,andforsharinginformationspecificallyaboutindividualyouthorfamiliesservedbythecollaborative.EachofthestrategieslistedinTable11(below)representamethodCMPsusetoenhancemoresystematicaccesstodataaboutcollaborativeactivitiesandservicesprovidedtoclients.Table11.CMPinformationsharingstrategies
InformationSharingStrategy(n=28)
Formaldatasharingagreementinplace. 13 (46%)Cross‐agencyaccesstodataforservicedelivery. 9 (32%)Cross‐agencyaccesstodataformeasurement/evaluation. 6 (21%)
NumberofCMPs
Reporting(%)
Theexistenceofaformaldatasharingagreementindicatesthat,tosomedegree,CMPpartnershavebeenabletoarticulateandformallydocumenttheirexpectationsofoneanotherwithrespecttothehandlingandsharingofdataaboutthecollaborativeanditsclients.Whilethirteen(13)CMPsindicatedhavingformaldatasharingagreementsinplace,thedegreeofspecificity,detailandbreadthoftheseagreementsvariessignificantlyacrossthesesites.Inmostinstances,itappearsthattheagreementsreferencedbyprojectsareembeddedinotherdocuments,suchasreleasesofinformationorCMPMemorandaofUnderstandingwiththestate,thataddressarangeofissuesbeyonddatasharing.Thesematerialsdonottypicallyaddresscross‐agencydatasharingwiththedegreeofdetailappropriateforsharingsensitiveclient‐levelinformationacrossCMPpartners.Asshowninthetableabove,nine(9)CMPsindicatedthattheyprovidecross‐agencyaccesstodataforpurposesofservicedelivery.Forexample,oneCMPreportedthat“sharinginformationacrossagencieshasbeeninvaluable.Weoftenaredealingwiththesamefamiliesandhavenoideathattheyareaskingforservicesfromotheragencies.”Again,thestructureandformalityoftheseprocessesvaries.Ingeneral,theseprocessesrelyonexistingdatabases,maintainedeitherbyalocalagencypartnerorastate‐levelagency.Mostfrequently,cross‐agencydataaccessisprovidedforspecificindividualsassociatedwiththeCMP(e.g.,theCMPCoordinatororISSTstaff)andforselectdatasystems,mostoftenTrails.Further,mostCMPsdescribedcross‐agencyaccessbetweentwo,ratherthanall,partneragencies.
32
CMPsalsodescribedthesystemsanddatatheyusetosupportprogramevaluationandmeasurement.Theseeffortsandrelateddataarelikelytohelpsupporttheassessmentofperformanceandtheidentificationofopportunitiesforimprovement.Onlyaboutone‐fifthofCMPs(6)reportedprocessesthatsupportcross‐agencyinformationsharingforthispurpose.However,on‐goingworkbytheSSCandinfrastructuredevelopedaspartofthestatewideevaluationhavecreatedsignificantopportunitiestoexpandthetrackingandsharingofdatatomeasureCMPperformance,asdiscussedfurtherinthedescriptionofthestatewidemeasurementplan(page55).Finally,nearlyallCMPs(27)reportedinformalmethodsofdataandinformationsharing,suchastelephoneoremailcommunication.Whilethesemethodsplayanimportantroleincoordinatedserviceplanninganddelivery,theymaybeinsufficienttosupportallCMPdataandinformationsharingneeds,particularlygiventhecomplexandfast‐pacednatureofCMPwork.
Localhighlight:CrossoverYouthPracticeModelOneCMPisimplementingtheGeorgetownUniversityCenterforJuvenileJusticeReformCrossoverYouthPracticeModel,whichitcitesasakeyapproachinitseffortstoreduceduplicationandfragmentation.Specifically,theCMPreportedthat“theCrossoverYouthPracticeModelisusedtoidentifyyouthinvolvedintheChildWelfaresystemassoonastheyarearrestedandscreenedbySB94.MoreformalizedjointplanningisconductedbytheJuvenileJusticesystemandtheChildWelfaresystem.DiversionalsoaddressesCrossovercasesandworkstonotduplicateexpectations.TheCourthasbeenusingthe…ISSTtoassistinthedevelopmentofmanagementplanswhenyoutharedeemedincompetent.OneISSThasincreasedthenumberoftruancyyouthstaffedinordertoreduceunnecessaryreferralsto[theDepartmentofHumanServices]whenservicescanbeprovidedbypartneragenciesorbarrierscanbeproblem‐solvedcreativelywiththefamilyandassistanceprovided.”PertheCenterwebsite,theCrossoverYouthPracticeModelbringstogetherresearchandpracticelearningaboutstrategiestoserveyouthengagedinboththechildwelfareandjuvenilejusticesystems.ConsistenttheCMPprinciples,themodelemphasizes“thefollowingpractices:thecreationofaprocessforidentifyingcrossoveryouthatthepointofcrossingover,ensuringthatworkersareexchanginginformationinatimelymanner,includingfamiliesinalldecision‐makingaspectsofthecase…andmaximizingtheservicesutilizedbyeachsystemtopreventcrossoverfromoccurring.”(http://cjjr.georgetown.edu/pm/practicemodel.html)
33
E. EffectivenessofCMPsinreducingduplicationandeliminatingfragmentation
TheCollaborativeSuccessSurvey(seelinktoadditionalinformationinAppendixB)includedmultipleitemstoassesstheeffectivenessofCMPeffortstoreducebothduplicationandfragmentation.SurveydataindicatethatIOGmembersperceivetheirCMPcollaborativetohaveledtoreductionsinduplicationandfragmentationacrossallmeasurementareas,asshowninFigure4.IOGmembersassignedanaverageratingof4.5orhigheroneachitem,indicatinghighperceivedsuccessinstreamliningandintegratingservices,andprovidingmorecomprehensiveservicesthataredeliveredbyfewerprofessionals,thusreducingduplication.
Figure4.Effectivenessofeffortstoreduceduplicationandfragmentation
1 2 3 4 5 6
Fewerprofessionalsservingonefamily
Morecomprehensiveserviceplans
Increasedcontinuityofservices
Greaterintegrationofservices
Streamlinedserviceprovision
Reducedduplicationofservices
Note:Itemsareratedona6pointscale(1=stronglydisagree,2=disagree,3=disagreemorethanagree,4=agreemorethandisagree,5=agree,6=stronglyagree).
F. Challengesassociatedwithduplicationandfragmentation
Despiteprogresstowardreducingserviceduplicationandfragmentation,manyCMPsreportedsignificantbarriersthatimpedeadvancementinthisarea.First,someCMPsexpressedfrustrationaboutthelackofconnectionbetweendifferentstate‐leveldatabases,suchasTrailsandEclipse.Becauseclient‐levelrecordscannoteasilybelinkedacrossthesedatabases,CMPandpartneragencystaffmaystruggletoobtaincompleteinformationaboutyouthbeingservedacrosssystems.CMPsindicatedthatestablishingconnectionsacrossthesedatabasescouldenablethemtomoreeffectivelyidentifyyouthappropriateforCMPservices,aswellasassessinstancesinwhichintegratedplanningmightbebeneficial.CMPsalsociteddifficultiesaccessingdatainspecificsystems(e.g.,thosemaintainedby
34
schoolsandprivateproviders)andensuringaccesstoexistingdataforindividualpersonnel(e.g.,enablingISSTfacilitatorstoaccessTrailsdata).Second,manyCMPsdescribedbarriersaroundinformationsharingandconfidentiality,specificallynotingchallengesindevelopingacommonreleaseofinformationthatmeetsallfederalandstatelawsaswellasthepoliciesofindividualpartneragencies.Finally,someCMPsindicatedthattheeconomicdownturnhaslimitedtheresourcesavailabletosupportinformationsharingandresultedinstaffturnoverinpartneragencies.WhereassignificantprogresshadbeenmadebyCMPstoreduceduplicationandfragmentationofservicesacrosstheirlocalsystems,furtheropportunitiesexisttotargetthislegislativegoal.Currentstrategiesincludedevelopmentofsharedformsandintegratedservicedeliveryplans,informationanddatasharingpractices,andjointly‐determineddivisionofresponsibilitiesforpaymentandserviceprovision.Securingsharedaccesstokeydatasourcesacrossagenciesremainsanimportantareaforcontinuedworkatthelocalaswellasstatelevelinordertoensureintegratedplanningcanoccur.Inaddition,itwillbeimportantmovingforwardforallCMPstodevelopmulti‐agencyconsentforms,ifthesearenotalreadyinplace,aspartofbroadservicemanagementeffortsaswellastoensurethatclient‐levelinformationcanbesharedacrossCMPpartners.Finally,developmentofasingular,integratedserviceplanthatcanbesharedamongprovidersisanotherareaforcontinuedexplorationinordertotargetreductionsinduplicationandfragmentationofservicesformulti‐systemfamilies.
35
CMPeffectonservicesforyouthandfamilies
“Historically,smallruralcommunitieshavehadtorelyoncollaborationtomakelimitedfundinggoalongway.Byformalizingtheprocessandaddingfunding,ithasallowedmanymoreservicestostayinplacethatwouldhavebeeneliminatedduetotheeconomicclimate.Theseservicesareabsolutelycriticalinlowsocio‐economicareaswhereneedyfamiliesarealargeportionofthepopulation.”TheCMPhasledto“afocusonEBPs[Evidence‐basedPrograms]asasystemandstartingtocreateacommonlanguageonwhatweareaboutandwhatwewanttoaccomplish.”
LegislativeGoalIII:Increasethequality,appropriateness,andeffectivenessofservicesdeliveredtochildrenorfamilieswhowouldbenefitfromintegratedmulti‐agencyservices.Thethirdlegislativegoaloftheinitiativeistoimproveservicesprovidedtofamiliesengagedwithmultipleagencies.Inresponse,CMPshavedevelopedandimplementedanumberofstrategiesthataredesignedtoimpactservicequality,appropriateness,andeffectivenessformulti‐systemfamilies.Thissectionexaminessomespecificapproaches,includingtheuseofevidence‐basedorevidence‐informedmodels,andstrategiestoidentifyexistingservicegaps.Weconcludethesectionwithadiscussionoftheevidenceofperceivedeffectivenessinthisarea.DataandinformationpresentedthroughoutthissectionweredrawnfromCMPannualreportsandsurveydataoncollaborativeeffectivenessfromthe2010‐2011fiscalyear.
A. Servicequality
Yearoneevaluationfindingsindicatedthatstakeholders,byandlarge,believethatlocalCMPshadimprovedthequalityanddeliveryoflocallyprovidedservices.Inthesecondyear,CMPscontinuedtomakeprogressinthisareawith17citingimprovementsinservicequalityasaprimarysuccessthisyear.Specifically,IOGshaveundertakeneffortstoestablishorrefinetheirISSTmodels,introducenewprograms,focuseffortstowardidentifiedtargetpopulations,ensureearlycasecontact,andprovideongoingmonitoringandfollow‐upwithfamilies.WithintheISSTstructure,CMPshaveaddressedservicequalitythroughtheimplementationofevidence‐based,orevidence‐informed(i.e.,afederalagencyisreviewingtheevidencetodetermineiftheprogramcanbedesignatedasevidence‐based)servicemodels.YearoneevaluationresultsindicatedthatHighFidelityWraparoundviiiandTeam/FamilyDecisionMakingixweretwocommonlyimplementedmodels.Thisyear,CMPswereaskedtoprovideadditionalinformationabouttheirchosenevidence‐basedorevidenced‐informedmodels.
36
Communityneedsandtheuseofevidence‐basedservices
“AsaresultoftheCMPtherehasbeenaformalvenueforIOGmemberagenciestomeet,discussissuesandmakecollaborativeplanstoaddressissuesrelatedtochildren,youthandfamilieswhoarehighriskofincreasedinvolvementwiththeIOGmemberagencies.ThiscoordinationhasalsoledtoadedicationtocollaborationthatextendsbeyondtheIOGandintotheotherarenasthatouragenciesintersect.Therehavebeenmoreprotocolsestablishedthatencouragecollaborationandmulti‐agencyassessment.Thecollaborativeapproachhasalsoledtoassessingcommunityneedsanddevelopingprogrammingandservicestomeettheneedsofchildren,youth,andfamiliesthatisevidence‐basedorevidenceinformed.”
TheWraparoundmodelcontinuestobeoneofthemostcommonlyimplementedevidence‐basedISSTapproaches(reportedby20CMPs)andisdemonstratedeffectiveinintegratingservicedeliveryandenhancingquality.Table12reflectsthenumberandproportionofCMPsreportingtheuseofaWraparoundmodel,includingwhethertheyimplementedHighFidelityWraparound;whethertheirWraparoundmodelisofferedtoalloronlyselectCMPyouthorfamilies;andwhethertheycollectevaluativeinformationreflectingthedegreeofimplementationfidelity.
Table12.CMPuseofthewraparoundmodelinservicedelivery
WraparoundPractices
CMPsUsingWraparound(n=27) 20 (74%)CMPsUsingHighFidelityWraparound(n=20) 7 (35%)CMPsOfferingWraparoundtoAllChildren(n=20) 14 (70%)CMPsCollectingDataonWraparoundImplementationQuality(n=20) 7 (35%)
NumberofCMPs(%)
Asnotedabove,themajority(74%)ofCMPsreportedusingaWraparoundmodel,withaboutathird(35%)ofthoseusingthemorehighlystructuredHighFidelityWraparoundmodel.Themajority(70%)ofCMPsofferingWraparounddosoforallyouthorfamiliesserved.Ingeneral,thesix(6)CMPsnotofferingWraparoundtoallyouthorfamiliesindicatedthattheyselectyouthorfamiliesforWraparoundbasedonthecomplexityofthecase,withmorecomplexcasesreceivingpriority.WhilewidespreaduseofaWraparoundmodelreflectspositivelyonthequalityofserviceprovision,itisnoteworthythatonlyaboutone‐third(35%)ofthesecountiestrackdataonthequalityoftheirimplementationefforts(i.e.,fidelity).Additionally,nineteen(19)CMPsreportedtheuseofotherevidence‐informedprogramsandpractices,andtwo(2)reportedthatuseofsuchprogramsisapriorityforthefuture.WhentheydonotserveasthemainISSTmodel,evidence‐basedorevidence‐informedprogramsarefrequentlyofferedthroughoneoftheIOGpartneragenciestofamilieswho
37
haveparticipatedintheISSTprocess(i.e.,asasuggestedinterventionemergingfromtheintegratedplan).Consistentwithfindingsfromlastyear,CMPshighlightedtheuseofnationallyrecognizedpreventiveinterventionssuchasMulti‐SystemicTherapyxorFunctionalFamilyTherapyxi,whileothersimplementprevention‐focusedprogramssuchasLifeSkillsTrainingxiiortheIncredibleYearscurriculum.xiii
B. Appropriatenessofservices
Appropriatenessofservicesreferstothefitbetweentheservicesprovidedandtheindividualfamilybeingserved.Servicescanfailtobeappropriatewhentheydonotconsiderthefullrangeoffamilycircumstancesoraccountforfamilyinteractionswithotheragenciesandpotentiallycompetingdemands.Meaningfuleffortsintheareasoffamilyengagement,treatmentmatching,andservicecoordinationhavebeenshowninresearchtoincreaseratesoftreatmentcompletionandtoimproveoutcomes.xivAsdescribedintheyearoneevaluationreport,theadoptionofISSTsisacentralstrategyforpromotingappropriatenessofservicedeliverywithinaCMP,particularlywhentheISSTsuccessfullyengagesfamiliesininterventionplanningandserviceselectionefforts.Thisyear,CMPswereaskedtofurtherdescribeeffortstoensuretheappropriatenessofservicesfortheyouthandfamiliestheyserve.Table13belowpresentsalistofkeystrategiesbywhichserviceappropriatenessmightbeassessedandenhanced,alongwiththenumberandproportionofCMPsreportingeach.
Table13.CMPstrategiestoenhanceappropriatenessofservices
Strategy
IdentifiedserviceareathatiscurrentlynotbeingaddressedthroughinformalneedsassessmentwithCMPpartners(n=27)
24 (89%)
IdentifiedserviceareathatiscurrentlynotbeingaddressedthroughformalneedsassessmentwithCMPpartners(n=27)
16 (59%)
Implementedanewprogramormodelthatspecificallytargetsthepopulationandserviceneed/gapthatwasidentifiedbyIOG(n=27)
20 (74%)
Implementedorenhancedservicestobemoreculturallyappropriate/culturallycompetent(n=27)
11 (41%)
Other(n=25) 5 (20%)
NumberofCMPs(%)
38
Asreflectedabove,thevastmajority(24;89%)indicatedusinginformalneedsassessments,suchasdiscussionsofexistingservicegaps,asameanstoensuringserviceappropriateness.Thenextmostcommonstrategy,reportedby20CMPs,wastheimplementationofnewprogramsormodelsthatareintendedtofillanidentifiedservicegap.SpecificprogramsvaryamongCMPsbutincludedthecreationofanewISST,anewmentoringprogram,enhancedcasemanagement,andayoungparentsupportgroup.Adifferentareaoffocusreflectedeffortstoimprovetheculturalfitofservicestoidentifiedfamilies.ElevenCMPsreportedworkingtowardimprovementsinthisareaandexamplestrategiesincludedutilizationoftranslationservices,supportoflocalgroupsfocusedonculturalcompetency,andtheprovisionofculturalcompetencytrainingstostaff,eachofwhichwasreportedby3CMPs.
C. Effectivenessofservices
TheCollaborativeSuccessSurvey,describedearlier(seeAppendixBforfurtherinformation),includedaseriesofitemsrelatedtoservicequalityincludingcommitmenttotheuseofevidence‐basedorevidence‐informedmodels,improvementsinservicequalitywithinindividualpartneragencies,andeaseofaccesstoservices.AsshowninFigure5,respondentsreportedthatCMPimplementationhashadapositiveeffectontheoverallqualityofservicesprovidedtofamilies,assigninganaverageratingofgreaterthan4.5onasix‐pointscaleoneachoftheseitems.
Caseexamplesofeffortstoenhanceappropriatenessofservices
OneCMPreportedtherecentcompletionof“atwo‐yearreviewofthecontinuumofservices,includingquantitativeandqualitativeassessments,andacomprehensiveandinclusivecollaborativeprocess.Thisresultedinthedevelopmentofanewcontinuumofservicesmodelthatwillbeimplementedoverthenext2‐3years.”“[TheCMPuses]athree‐tieredmodelwhereeachtierrepresentsthelevelofreferredclient/familyriskandthenmatchthattotheappropriatemodel,levelofservice,andfundingallocation.Theideaistoimprovetheappropriatenessofservicesandtonotfloodlowriskfamilieswithtoomanyservicesandensureweareusingourincentivefundinginthemostappropriatemanner.”
39
Figure5.Effectivenessofeffortstoenhanceservicequalityandappropriateness
1 2 3 4 5 6
Improvedqualityofservices
Committedtoevidence‐based/informedmodels
Easierforfamiliestoaccessservices
Committedtoa"nowrongdoor"approach
Note:Itemsareratedona6pointscale(1=stronglydisagree,2=disagree,3=disagreemorethanagree,4=agreemorethandisagree,5=agree,6=stronglyagree).CMPscontinuetoimplementstrategiesfocusedontheidentificationandimplementationofservicesmeetinghighstandardsofquality,whilealsoensuringthatthoseservicesarewellmatchedtothestrengthsandneedsoftheyouthandfamiliesbeingserved.Additionally,theSSCandCMPportalprovideon‐goingopportunitiesforCMPstolearnaboutevidence‐basedprogramsbeingimplementedthroughoutthestate,aswellasthestrategiesothercommunitiesareusingtoensurethequality,appropriateness,andeffectivenessoftheirservices.Nonetheless,itwillbevaluableforCMPstakeholderstoexploreotherservicesandprogramsthatmaybewell‐alignedwithCMPgoals.Tothatend,evaluationactivitiesinyearthreewillincludecontinuedeffortstoidentifybestpracticesrelatedtobothcollaborationandservicedelivery,drawingupontheextantresearchliterature,aswellasCMPpracticetodate.
40
Cost‐sharing:Useofblendedandbraidedfunds
“[OurCMP]continuestoincreasethefrequencyofblended/braidedfundingforthepurchaseofspecificallyidentifiedservicesforyouth/familieswhichhasincreasedeffectivenessoftheintegratedplansforclients,aswellasincreasedthestrategicuseofmultiplefundingstreams,andreducingfragmentationofservicesbetweenagencies.Ithasalsoelevatedsomediscussionsandinformalagreementsaroundwhopaysforwhatandwhenasitrelatestoasystemicapproachratherthanagency‐specific.Thereisincreasedcooperationonprojectsbetweenpartneragencieswhenitcomestothepursuitorinvolvementingrants,whichhasalsosupportedmorecross‐systemtrainingofstaffincertainareas.”
LegislativeGoalIV:Encouragecostsharingamongserviceprovidersthatleadstocost‐reductionfortheservicesprovidedtochildrenandfamiliesinthechildwelfaresystem,includingthefostercaresystem,inthestateofColoradoThecollaborativemanagementapproachassumesthatdecreasesinserviceduplicationcombinedwitheffortstopreventdeeperandmorecostlyinvolvementinvarioussocialservicesystemswillleadtobetterfamilyoutcomesandnetcost‐savingsovertime.Infact,researchdemonstratesthatcostsaredrivenhigherbypoorlycoordinateddeliveryefforts.xvThedevelopmentofinteragencyprocessesthatsuccessfullyintegrateservicedeliveryshouldreducetheneedfor,andthecostsof,duplicatedcare.ThissectionoutlinesCMPs’approachestocost‐andresource‐sharing,andreportedperceptionsofsuccess.WefirstexploretheextenttowhichsitesallocateandshareresourcestosupportCMPoperationsandadministrativeneeds;payforexistingandnewservices;andensuresustainabilityofprograms.Wethenexaminereportedcostsavingsandre‐investmentoffunds.Thesectionconcludeswithadiscussionofthechallengesrelatedtocost‐benefitanalysisandtheinfrastructurenecessarytospecifycostmodelsacrosstheCMPinitiative.Datapresentedinthissectionwerederivedfrom2010‐2011fiscalyearannualreports.
A. Cost‐sharingapproaches
ThemanagementoffundsandthedevelopmentofeffectiveprocessestoshareoperationalandservicedeliverycostsamongagencypartnersarecoreactivitiesofIOGs.Althoughonepartneragencytypicallyservesasthefiscalagent(thisisthelocalDHSin69%ofCMPs),allocationdecisionsaremostoftenmadecollaboratively,bytheIOG(72%)oranIOGsubcommittee(24%).Specificmethodsofcost‐sharingaredescribedbelowandsummarizedbelowinTable14.
41
i.SharingcostsofCMPoperations
SitesreportedthatIOGpartners“frequently”or“always”jointlypaidforstaffingandadministrativecosts(25and20CMPs,respectively).ThisoftenisachievedthroughpoolingfundstosupportsalariesofIOGcoordinatorsorprogrampersonnel,andcontributingin‐kindresourcessuchasofficespaceandtrainings.Partneragenciesalsofrequentlysharefundinginordertodevelopnewprograms,thusexpandingservicecapacity.Table14.CostsharingamongCMPpartners
Always 12 (46%) 8 (30%) 3 (11%)Frequently 13 (50%) 12 (44%) 11 (41%)Sometimes 1 (4%) 6 (22%) 10 (37%)Rarely 0 (0%) 1 (4%) 1 (4%)Never 0 (0%) 0 (0%) 2 (7%)
FrequencyStaffingCosts
(n=26)
AdministrativeCosts(n=27)
CostsofNewPrograms(n=27)
ExtenttoWhichCMPPartnersShareCostsNumberofCMPs(%)
ii.Sharingcostsofservices
CMPsemphasizedthatcost‐andresource‐sharingisanintegralpartoftheISSTprocess.Asnotedearlier,themajorityofCMPsindicatedthatduringISSTmeetings,agenciesjointlyagreeonwhowillpayforservicesproposedintheinterventionplan.Agenciesmayagreetosharethecostofanexpensiveservice,ornegotiatetohaveeachagencypayforseparateservices.Asacomplementtothesecost‐sharingprocesses,someCMPsallocateportionsoftheiryearlyincentivefundingtopayforinterventionservices.Theseventeen(17)CMPswithavailabledatareportedspendingnearly$714,000onservicesprovideddirectlytochildrenorfamilies,whichrepresented29%oftheir2010‐11fiscalyearincentivefunds.
iii.Ensuringsustainabilityofservices
ManyCMPsareworkingtoexpandfundingandpromoteprogramsustainability.Toachievethis,IOGpartneragenciesoftensharegrantwritingresourcesinpursuitofexternalgrantopportunities.Inthe2010‐2011fiscalyear,10ofthe29CMPs(34%)reportedreceivingcompetitivefundsfromfederalagencies,stateagenciesandblockgrants,andprivatefoundations(seeAppendixCfordetailedinformation).Together,theseeffortsgarneredatleast$1,084,776inadditionalfunding.3CMPsoftenaresuccessfulintheseendeavorsbecauseoftheirstrongcollaborativerelationships.Theexistingpartnershipsareattractivetofundersbecausethefoundationneededforeffectiveimplementationis
3 This total is underestimated due to missing data on total amounts received.
42
alreadypresent.TheinfrastructurebuiltbytheCMPinitiativeisrecognizedasbeingakeymechanismtoimplementprogrammingacrossColorado.Forexample,CMPcommunitiesareafocusofastatewideplanninginitiativetoensurethatcountiesembracetheSystemsofCareapproachtoservingchildrenandfamilieswithseriousemotionaldisturbances.
B. Costreductions,potentialcostsavings,andreinvestmentoffunds
Anunderlyingassumptionoftheinitiativeisthatcollaborativeeffortswillleadtocostreductionsandthuscostsavings,whichmaybere‐investedinlocalCMPactivities.However,precisemeasurementofthesecostcomponentsisdifficult,asitrequiresacalculatedestimateofthepotentialcostsforagivenindividualorfamilyhadCMPeffortsnotbeenprovided.Whileoutcomesmaybeimproved,itisdifficulttodeterminewhetherthesewereachievedatlowercost.Moreoverthelackofstandardizedoutcomesandmeasurementprotocolsacrossprojects,asevidencedinthefirsttwoyearsoftheevaluation,preventstheassignmentofpotentialsavingsestimates.
Inthe2010‐2011fiscalyear,onlythreeCMPsreportedhavingaformalprocesstomeasureactualcostsavings.Twelve(12)providedroughestimatesofsavingsbasedupontheirperceptionsoftheextenttowhichservicesmayhaveledtoreductionsinthenumberofchildren,andtheamountoftimespentinout‐of‐homeplacements,includingfostercare,therapeuticresidentialandpsychiatriccare,andcommitmentanddetention.ThesetwelveCMPsreportedatotalof$5.9millionofsavingsthatwerereinvestedinCMPefforts.
CMPsdescribedothersourcesofperceivedcostreductions,butwereunabletoquantifytheassociatedcostsavings.Theseinclude:
Fewerfamiliesenteringthechildwelfaresystem Reductionsintruancyrates(leadingtopresumedlowercostsineducation,child
welfare,andjuvenilejusticeprograms,andincreasedrevenuesforschools) Reductionsinthelengthofmentalhealthtreatmentandinpatientstays Improvementsincoordinationandaccesstoneededcareandfundingsourcessuch
asMedicaidandCHP+ Successfulpreventionofcostlyservicesthroughtargetedprograms,including
pregnancyprevention.
43
ArecommendationfromtheyearoneevaluationwastoworktowardmethodstotrackcostsofCMPservicesmoresystematically.Inthecurrentyear,CMPsbasedestimatesofanISSTstaffingontimespentbyprovidersinanaveragemeeting.Amongthe21CMPsthatrespondedtothisitem,theaveragecostwas$323,witharangeof$110to$800.Thelargerangeofestimatedservicecostsistobeexpected,giventhediversityofISSTmodelsandservices.Thatis,someCMPsstructuretheirISSTssuchthatbrief,frequentmeetingswithsmallernumbersofparticipatingprovidersareheld,whileothersmayholdonelonginitialISSTplanningmeetingwithalargernumbersofproviders.Somesitesindicatedthatitwasdifficultorunworkabletocalculateameaningfulaveragecost,giventhattheirserviceplanningmeetingsareadaptedtotheneedofeachfamily.
i.ReinvestmentoffundsTotheextentthatCMPsdoexperiencecost‐savings,thelegislationrequiresthemtore‐investthosefundsintoCMP‐relatedoperations.Twenty‐one(84%)ofthe25CMPsreportingthesedatainthe2010‐2011fiscalyeardescribedtheiractualorplannedreinvestmentoffunds.Asreported,fundswerefrequentlyreinvestedintheprovisionofadditionalsupporttoexistingornewservices(16CMPs),tosupportcoordinatorsalaries(4CMPs),andtobuildneededinfrastructure(e.g.,databasedevelopment;3CMPs).Otherlessfrequentlymentionedusesincludedcreating“emergency”accountstoassistfamiliesinpayingfordirectservicesorimmediateneedsandpayingforexternallocalevaluationservicestoassessperformanceonkeyoutcomes.
C. Challengesrelatedtomeasuringcostsandbenefitsattheinitiativelevel
Testingthecost‐savingshypothesisassumedinlegislationiscomplicatedgiventherangeanddiversityofexistingprojects,processesandoutcomes.Below,weoutlinekeydifficultiesassociatedwithquantifyingcostsandbenefitsforCMPefforts.Wethendescribe
Localhighlight:MeasurementofcostreductionsOneCMPestimatescostsavingsusingapre‐andpost‐servicedesign.Theycollectdataretrospectivelyonactualcostsofservicesincurredbyfamiliesduringtheone‐yearperiodpriortoenrollmentintheirISST.BaselinecostsincludeanyDHSCoreServices,mentalhealthtreatmentcoveredbyMedicaid,DYCdetention,andanydaytreatmentcosts.ThetotalbaselinecostsarecomparedwiththeactualcostofimplementingtheirWraparoundmodelservices.ThisCMPreportedestimatedsavingsof$236,000thisyear.ThismethodisnoteworthyasitreliesuponactualcostsamongfamiliesservedbytheCMP;thus,totalsavings,whilestillanestimate,couldbereasonablyattributedtoCMPservicedelivery.
44
thesupportsneededtomovetowardthespecificationofoneormorecostmodelsfortheinitiativeinthefuture.Initssimplestform,cost‐benefitanalysissumsthevalueofthebenefitsaccruingfromasetofactionsandthensubtractsfromthis,thesumofthecostsassociatedwiththoseactions.Thus,inordertoconductacost‐benefitanalysis,itisnecessarytoassignmonetaryfigurestobothsidesofthisbasicequation:costsandbenefits.ThisischallengingfortheCMPinitiativeforavarietyofreasons.First,itisdifficulttoassignmonetaryvaluesonthecostsideoftheequationasmostCMPeffortsdonothavesufficientlyclearprogrammodelsand/orlacktherequiredlevelofprecisionneededtospecifycosts.Thatis,modelshavenotbecomestandardenoughorimplementedwithsufficientfidelitytoenableaccuratecostassignmenttoefforts.Becauseitisalsocriticaltoassignastandardcostregimenacrossmultiplesites,thisvariabilityproblemiscompoundedwhenseveralCMPsareconsideredforanalysis.Thebenefitssideissomewhatmorecomplicated.ItisassumedthatCMPeffortsresultinoutcomesthatareachievedatlowercostbecausewithouttheCMP,serviceswouldhavebeenmoreexpensiveduetoserviceduplication,fragmentation,andaprolongedneedforcostlyservices.Inordertoassignmonetaryvaluestothesebenefits,onemusthypothesizewhatwouldhavehappenedwithoutCMPservices;thatis,whatthelikelytrajectoryofnegativeoutcomesfromagivensetofproblemsmighthavebeenwithoutCMPinterventionandwhattheassociatedcostswouldhavebeen.IfthetotaloftheseestimatedcostsexceedsthetotalcostofCMPcollaborativeprocessesplusprovidedservices,thentheCMPwouldbeassumedtohavehadanetfinancialbenefit.However,specificationofthesevarioushypotheticalcostsisverydifficultbecausewedonothavereliablemethodsofpredictinglong‐termchildandfamilyoutcomes.Shouldtheinitiativeseektodevelopacost‐benefitmodelinthefuture,manyofthebarriersabovewouldneedtobeaddressedthroughthefollowing:
greateradoptionofapreciseandstandardprogrammodel increasedimplementationfidelity agreementoncorrespondingoutcomes
However,beforemovinginthisdirection,theinitiativewouldneedtoconsidersomeimportanttrade‐offs.Specifically,theCMPeffortplacesahighvalueonlocaldecisionmakingandthediversityofapproachesisoneofitsstrengths.Whilethepromiseofarigorouscost‐benefitapproachmayseemliketheidealevaluationmodel,implementing
45
requiredchangeswouldnecessitategreaterstandardizationacrosssites,andmaycomeattheexpenseoflocalinvestmentandsupportfortheCMP.TheplantouniformlymeasureaselectsetofservicedeliveryandoutcomeindicatorsacrossallCMPsinthe2011‐2012fiscalyear(mentionedaboveanddiscussedindetailinthefollowingsection)movestheinitiativeastepclosertomeasuringcostsandbenefits.Withcontinuedevaluationeffortsinthisarea,itishopedthatovertime,therewillbeanopportunitytospecifyacost‐benefitmodelandconductrelatedanalyses.Insummary,theevaluationrevealedthatlocalpartneragencieshavedevelopedclearprocessestosharecostsofoperationsandservices.Inmanycases,localCMPshaveeffectivelyleveragedtheircollaborativeeffortstoobtainadditionalfundingtosustainand/orgrowtheirprogramsandservices.However,itisdifficulttoascertaintheextenttowhichCMPstructuresandprocesseshaveledtocostreductionsandcostsavings,giventhedifficultiesinmeasurement.Onesitewashighlightedfortheirmethodofaccountingforpre‐andpost‐interventionservicecostsindeterminingcostsavings.However,thelocalresourcesneededtocollectretrospectiveservicedatamaybeprohibitiveforCMPsservinglargenumbersofchildrenandfamilies.Thus,whilerepresentingapromisingstrategytomorepreciselyassesscostsavingsatthelocallevel,thismethodmaynotbefeasibletoapplyacrossallCMPs.Animportantnextstepfortheinitiativewillbetodevelopfeasiblemethodstomorespecificallyassesscosts(e.g.,costtemplates).Inaddition,evaluationactivitiesinyearthreewillexplorethedevelopmentofuniformmethodsofmeasuringcostsavingsassociatedwithspecificCMPprocessesand,totheextentpossible,specificoutcomes.
46
LegislativeGoalV:Leadtobetteroutcomesandcost‐reductionfortheservicesprovidedtochildrenandfamiliesinthechildwelfaresystem,includingthefostercaresystem,inthestateofColorado.ThefifthandfinalgoaloftheCMPinitiativeistoimproveoutcomesforchildrenandfamilieswhorequireservicesfrommultipleagencies.Itassumesthatinteragencysystemimprovementsresultinginmoreefficientandeffectiveservicedeliveryalsoshouldleadtomoresuccessfulandpermanentoutcomesforchildrenandfamilies.ThissectionfirstdescribesthecomplexnatureofCMPservicepopulationdefinitionsandreportsestimatesofthenumberofchildreneligiblefor,andservedby,CMPs.Wethenexplorewhattheevaluationrevealedaboutdefiningandmeasuringoutcomes,anddiscusscurrentlimitationsinassessingthestatewideimpactoftheCMP.WepresentfindingsfromlocalCMPperformancemeasuresduringthe2010‐2011fiscalyearand,whereappropriate,discusstheseinlightoffindingsfromlastyear.Thissectionconcludeswithadetailedoutlineoftheupdatedevaluationplanwhichhelpstoaddresssomeoftheidentifiedlimitations.Datasourcesforthissectionincludemid‐yearandannualreports.
A. DefiningCMPtargetpopulations
Currently,itisdifficulttoobtainprecisenumbersofindividualsandfamilieswhoareeligiblefor,andservedbyCMPs,bothwithinandacrosssites.Accordingto24‐1.9‐102(2)(c)CRS,eachMemorandumofUnderstandingissupposedtoincludeafunctionaldefinitionof“childrenandfamilieswhowouldbenefitfromintegratedmulti‐agencyservices."However,thereisnostandarddefinitionofeligibilityforCMPserviceefforts.Asaresult,CMPshavespecifiedvaryingdefinitionsoftheireligibleandservedpopulations,whichrangeinbothbreadthandspecificity.Examplesinclude:allchildrenwithinthecounty;allchildrenwithopencasesintheircounty’schildwelfaresystem;allchildreneligibleforservicesinanyoftheirpartneragencies;allchildrenreceivingservicesinatleasttwopartneragencies;andallchildren“atrisk”forinvolvementwithpartneragencies.Additionally,methodsusedtoidentify,countandmanageinformationonyouthandfamiliesalsovary.FewCMPs,forexample,havedatamanagementsystemsthattrackindividualsorfamiliesservedthroughtheprogram.Further,manyCMPsrelyoncounty‐,district‐,orstate‐leveldatabasesorpartneragencydatasystemstoobtaincounts;however,thesecountsoftenincludelargerpopulationsthanthoseservedthroughlocalCMPefforts.Finally,someCMPscountindividuals,otherscountfamiliesandsomecountbothastheirunitsofmeasurement,andcountsmaybeunduplicated(eachindividualorfamilycountedonce)orduplicated(eachindividualorfamilyrepresentedmorethanonce)
47
inthetotal.Thevariationinhowindividualsandfamiliesaredefinedandcountedatthelocallevelmakesitdifficulttodeterminetheextenttowhichlocalprojectsareservingthetargetpopulationasoutlinedinthelegislation(e.g.,childrenandfamiliesinvolvedinmultiplesystemsofcare).Takentogether,thepresentcountsofeligibleandservedpopulationsdiscussedbelowshouldbeinterpretedwithcautionandarelikelyoverestimated.
i. EstimatesofCMPeligibleandscreenedpopulation
Inthe2010‐2011fiscalyear,therewasanestimated70,000individualsconsideredeligibleforCMPservicesandapproximatelyhalf(30,000)ofthatpopulationwasreportedlyscreenedforservicesacrossallCMPs.CMPsestimatedthefollowingnumbersoftotalindividualsserved,calculatedatdifferentlevelsofservicedelivery(allincludeduplicatedcases):
19,600individualswhoparticipatedinlocalservicesoreffortsthatweredirectlyassociatedwiththeirCMP.TheseeffortsreflecttotalsfromallCMPsandcouldincludeservicespaidbyCMPincentivefunds,servicesdonatedin‐kindorprovidedbypartneringagencies,allthoseservedbyISSTs,andanylarge‐scalepreventionorinterventioneffortsthatweresupportedbytheirIOG.
7,000individualswhoreceivedservicesthroughthelocalISSTprocess.Thesereflecttheindividualswhoparticipatedinamulti‐agencyproviderstaffingmeetingsacrossallCMPs.
5,300individualswhoreceivedservicesthatwerefunded,atleastinpart,byCMPincentivefunds.Thistotalnumberwascalculatedacross19CMPswhereindividualsparticipatedinprogramswhichtheIOGfinanciallysupportedwithincentivefunds.FourCMPsindicatedthatnoindividualsreceivedservicessupportedbyincentivefunding(i.e.,partneragenciesprovidedin‐kindservices),andanotherfourhadnotyetreceivedincentivefunding.
B. DefiningCMPoutcomes
Eachyear,localIOGpartnersdescribetheirplantoassessprogressonkeyoutcomesintheirMOUs.Theyspecifycomponentsoftheirmeasurementplan,whicharedefinedbelow:
Targetedoutcomedomain.LocalCMPsarerequiredtoassessperformanceinthefouroutcomedomains(i.e.,childwelfare,juvenilejustice,education,andhealth/mentalhealth).Sitesmayalsoassessperformanceinotherdomainssuchasdomesticviolence.
48
Targetedoutcomes.CMPsindicatewhatgeneraloutcomestheyaremeasuringwithinthesedomains(e.g.,reducetruancy,increasestabilityofout‐of‐homeplacements).
Performanceindicators/measuresandgoals.Foreachoutcome,sitesdefinetheindicator(specificmeasure)thattheyemploytoassessprogressontheoutcomeandstatetheirannualtargetgoal(e.g.,increasetherateofCMP‐servedyouthwhoremainintheirhomesafterreceivingCMPservicesby2%).
Atthebeginningoftheinitiative,theSteeringCommitteespecifiedalistofapproximately20outcomesreflectingkeygoalsoftheCMP(e.g.,reduceout‐of‐homeplacements,reducerecidivism).Overthepastsixyears,however,thenumberandvarietyofoutcomesandrelatedperformanceindicatorsassociatedwiththeseoutcomesexpandedgreatly.Aspartoftheformativeevaluation,outcomesandperformanceindicatorswereexaminedeachyeartoassesstheextentofstandardizationacrossCMPs,aswellassimilaritiesinmeasurementanddatasources.ItwasdeterminedinyearoneoftheevaluationthatCMPsreportedonover150differentperformanceindicatorsandgoals.Inthesecondyear,thereviewrevealedthefollowing:
Numberofperformanceindicators/goals.Onaverage,eachCMPassessed7goals,althoughafewCMPsmeasuredtwentyormore.
Goalsbydomain.The28CMPswithatleastoneyearofdataproposed198performancegoals,distributedalmostevenlyinchildwelfare(49),juvenilejustice(49),education(49),andhealth/mentalhealthdomains(42).Ninegoalswereinotherareas,includingdomesticviolence.
Processversuspopulation‐levelindicators.Two‐thirds(134)ofstatedgoalsinvolvedpopulation‐levelindicators(e.g.,assessingchildandfamilyoutcomes),while64involvedprocessindicators(e.g.,improvementsinservicedelivery).
Definitionsofperformanceindicators.CMPsvarygreatlyintheirdefinedindicators.Despitesomecommonoutcomeareas,localCMPsmeasureverydifferentaspectsoftheseoutcomes,whichprecludedanyaggregateanalysisofeffects.
Populationlevel.Performanceismeasuredatdifferentpopulationlevels(i.e.,rateforentirecountyversusyouthservedthroughtheISST).
Datasources.CommondatasourcesincludetheCDHSTrailsdatabase,Eclipse/ICONstatejudicialdatabase,localschooldistrictdatabases,andDBHandColoradoClientAssessmentRecords(seeAppendixDforafurtherdescriptionofdatasources).
Thesesummaryfindingsshowthatingeneral,localsitestendtotracktheirperformanceinmanyoutcomes,utilizingmultipleindicatorswithinoutcomes.Thisisexpectedtosome
49
degree,giventhatCMPsemployavarietyofservicemodelsthataretargetedtolocalcommunityneeds,whicharediverse.Awide‐rangingmeasurementapproachallowsCMPstodemonstrateachievementsonseveralfronts.However,itmayalsocomeatacost,iflocalsites’attentionsandresourcesaredividedintermsofworkingontoomanyfrontsoroutcomesatonce.ThemoreCMPsarefocusedonasmallsetofcentraltargetgoalsanddedicatingresourcestodevelopingstrongindicatorsofthesegoals,theeasieritistodemonstrateaclearandpositiveimpactofefforts.Inaddition,with198differentperformanceindicators,allwithvaryingdefinitionsandmethodsofmeasurement,itiscurrentlyimpossibletoaggregateindicatorstoassessimpactacrossmultipleCMPsoratthestatelevel.However,tofacilitatedevelopmentofasummaryofachievementsacrosstheCMPinitiative,performanceindicatorswerecategorizedintothetenmostfrequentlyassessed(common)outcomes.Theseoutcomes,examplesofrelatedindicators,andthenumberofCMPsselectingtheseinyearsoneandtwoarepresentedinTable15onthefollowingpage.
50
Table15.Thetenmostcommonlyassessedoutcomes
Domain CommonOutcome ExampleIndicatorsNumberofCMPsassessingoutcomeFY09‐10 FY10‐11
ChildWelfare
Enhancestabilityofout‐of‐homeplacements
Reductionsinthenumberofmovesandlengthoftimethatchildreninout‐of‐homecaresettingsexperience
6 10
Preventout‐of‐homeplacement/increasereunification
Reductionsintherateofout‐of‐homeplacement,numberofchildrenservedthroughtheCMPwhosuccessfullyremainintheirhomes,reductionsintheaveragelengthofstayinfostercare
13 13
JuvenileJustice
Successfulcompletionofprobation
Increasingpositivedischarges,decreasingprobationrevocations,andincreasingthenumberofyouthnotviolatingcourtorders
17 16
Loworreducedusageofcommitment/detentionfacilities
MaintainingalowrateofAverageDailyPlacement(ADP)indetentionandlowratesoflong‐termplacement
5 7
Reduceormaintainlowratesofre‐offense/recidivism
Reducingthenumberofyouththatreoffendorareadjudicateddelinquent,orincreasingthenumberofyouthnotincurringnewchargeswhilereceivingCMPservices
6 8
Education
Increaseattendance/reducetruancy
Increasingyouthattendanceanddecreasingtruancyrates
18 25
Increaseorimprovestudentachievement
Improvementsinreadingscores,grades,andstandardachievementscores
7 4
Health/MentalHealth
Improvedleveloffunctioninganddecreaseinproblemseverity
Improvementsintheleveloffunctioningorproblemseverityofindividualsreceivingmentalhealthservices
12 14
Increasepreventionandtreatmentforsubstanceuse/abuse
Improvementinnumberofyouthremainingintreatmentprogramsforaspecificperiodoftime
5 1
Decreaseormaintainlowratesofhospitalization/inpatientservices
Lowerratesofinpatientcareandtherapeuticresidentialchildcarefosterplacementsandfewercasesofadmittingyouthintohigherlevelsofcare
3 5
i. Overallperformance
Wefirstexaminedthepercentofallperformancegoalsmeasuredacrosstheinitiative.CMPsreportedmeeting78%ofthe198totalperformancegoals.AsshowninFigure6below,successinmeetinggoalswashighinthreeofthefourdomains,withlesssuccessintheeducationdomain.
51
Figure6.Percentageoftotalperformancegoalsmet,bydomain
Note:N=198
ii. Performanceintencommonoutcomeareas
Inordertoexaminesummaryachievementsinspecificoutcomeareas,wedetailperformanceonthe103indicatorsreflectingthetencommonoutcomes,among26CMPsthathaveparticipatedintheinitiativeforatleasttwoyearsthusenablingassessmentofprogressfromFY2009‐10toFY2010‐11(seeTable16below).
Table16.Percentofperformancegoalsmet
Enhancestabilityofout‐of‐homeplacements 88%Preventout‐of‐homeplacement/increasereunification
80%
Successfulcompletionofprobation 69%Loworreducedusageofcommitment/detentionfacilities
86%
Reduceormaintainlowratesofre‐offense/recidivism
88%
Increaseattendance/reducetruancy 44%Increaseorimprovestudentachievement 60%Improvedleveloffunctioninganddecreaseinproblemseverity
60%
Increasepreventionandtreatmentforsubstanceuse/abuse*
100%
Decreaseormaintainlowratesofhospitalization/inpatientservices
57%
Health/MentalHealth
%GoalsMetFY2010‐11
Domain CommonOutcome
ChildWelfare
JuvenileJustice
Education
*Note:OnlyoneperformancemeasurewasassessedinthisoutcomeareainFY10‐11.
52
Improvingyouthoutcomes
“Themostsignificantchangeinoursystemhasbeentheeffectivecollaborationofpartneragencies.Thetruebeneficiariesofthishasbeenouryouth.Largelyduetothepartnership,wehavereducedthenumberofyouthcommittedtotheDivisionofYouthCorrections,we'vebeenabletoreducethenumberofyouthonprobation,placedinmentalhealthhospitalsorTRCCFlevelplacements.Theseyoutharebeingmaintainedintheirowncommunitiesandwiththeirrespectivefamilies.”
Ingeneral,themajorityoftheperformancegoalstargetingoutcomesinthechildwelfareandjuvenilejusticedomainsweremet,withmorevariableperformanceonoutcomesineducationandhealth/mentalhealthdomains.Afewinterestingdifferencesinproposedmeasurementandperformancebetweenthetwoevaluationyearswerenoted:
Preventout‐of‐homeplacements/increasereunification.Thenumberofproposedgoalsforthisoutcomegrewfrom15goalsinthe2009‐2010fiscalyearto21goalsinthecurrent(2010‐2011)year,suggestingthatthismaybeanareaofgrowinginterestandfocusamonglocalCMPs.
Successfulcompletionorterminationofprobation.CMPsmayhavetargetedadditionaleffortsinthisareaoverthepastyear,asalargershareofthesegoalswasmetthisyearthanlast(69%and44%,respectively).However,thesignificanceofthisshiftisuncertain,aslocalCMPsmayhavesetgoalswithlowerexpectationsinresponsetolastyear’spoorerperformance.
Increaseattendance/reducetruancy.Moreperformancegoals(25)weremeasuredinthisareacomparedtolastyear(18),suggestingthattheremaybegrowinginterestand/orneedtoprovidemulti‐agencyintegratedservicesinthisarea.
Increasepreventionandtreatmentforsubstanceuse/abuse.OnlyoneCMPmeasuredagoalinthisareathisyear,ascomparedtothefiveCMPsthatdidsointhepreviousyear.Thisdifferencemayhaveresultedfromchallengeswithdatacollection,particularlyinrelationtotimelycalculationofoutcomesforannualreporting.
Insummary,CMPeffortsresultedinreportedimprovementsinlocally‐definedperformanceindicatorsduringthefiscalyear.CMPsmetapproximatelythree‐quartersoftheirproposedgoals,thoughsuccessratesvariedsomewhatbyoutcome.AlthoughCMPsreporteddatareflectingtencommonoutcomes,thetargetpopulationsandspecificperformanceindicatorsforthoseoutcomesweredefinedandmeasureddifferentlyacrossCMPs.Thus,datacouldnotbemeaningfullyanalyzedacrosssitesbecausedirectcomparisonisneitherpossiblenorappropriate.Asdiscussedabove,issuesrelatedtooutcomemeasurementandanalysisledtothecreationofanOutcomesWorkgroup,discussedbelow.
53
C. Planformeasurementofstandardoutcomesinevaluationyear3
ThenewstatewideevaluationmeasurementplanwascollaborativelydevelopedbytheOutcomesWorkgroup,usingtheprocessoutlinedintheevaluationmethodssection(page9)andfurtherdetailedinacomprehensivereport(seeAppendixBforalinktothefullreport).ThemeasurementplanwasadoptedbytheCMPStateSteeringCommitteeinMarchof2011andimplementationbeganinJuly2011.TheplancentersonthecollectionandanalysisofaselectsetofindicatorsmeasureduniformlyacrossCMPsandaggregatedatthestatelevel.TheseindicatorsassesscoreCMPprocessesandoutcomes,includingthenumberofchildrenserved,theextentofserviceintegrationacrossagencies,andimprovementsinthefunctioningofchildrenandfamiliesservedbytheCMP.Additionalinformationaboutthespecificindicators,datacollectionprocesses,selectedoutcomes,anddataanalysisassociatedwiththisplanispresentedbelow.
i. Processmeasuresandmethodofdatacollection
ProcessindicatorswereselectedtoobtainanaccuratecountanddescriptionoftheCMPtargetpopulationandtotrackprogressoncoreservicecomponents.Theseprocessindicatorsare:
Numberofchildren/familiesreferredtoISSTs Numberandtypeofagenciesorsystemsthat:
o Familiesareinvolvedinatthetimeofenrollmento Participateinthefamily’sserviceplanningo ProvideservicestofamiliesasaresultoftheISST
Numberofcaseswherefamilymembersparticipatedinserviceplanning Numberofcasesinwhichanintegratedserviceplanwasdeveloped Numberandtypeofoutcomes,amongaselectionofoutcomesinthechildwelfare
andjuvenilejusticedomains,thatarealignedwithISST‐servedcasesAtwopagedatacollectionformtocollectitemsneededtomeasuretheabovelistedindicatorswascreated.Theformalsoincludesclientidentifyinganddemographicinformationandstateidentifiernumbersfromtwostatewidedatasystems(Trails,ICON/Eclipse).Tosupportlocal‐andstate‐leveldatatrackingandanalysis,anon‐lineclient‐leveldatabasesupportedbyEffortstoOutcomes(ETO)©softwarewasdevelopedspecificallyfortheinitiative.ETOwasselectedforanumberofreasons:itisawell‐establishedsocialservicedatatrackingsystem;itisflexiblesothatadditionaldataitemscanbeeasilyintegrated;andcustomreportscanbecreatedsothatlocalCMPscantrackandreportontheirserviceinformationthroughouttheyear.EachCMPhasestablishedprocessestocollectthedataon
54
childrenandfamiliesparticipatingininitialISSTmeetings.MostsiteswillenterthesedataintotheETOsystem;theotherswilltrackthesedataitemsintheirlocaldatabaseandprovideOMNIwiththedatafile.OMNIisprovidingtrainingandtechnicalassistanceforthesedatacollectionsystemsandwillbeconductingqualitycontrolchecks.
ii. Outcomesmeasuresandmethodofdatacollection
TheOutcomesWorkgroupidentifiedasetofindicatorsthatreflectcoreexpectedoutcomesoftheCMPinitiativeinthechildwelfareandjuvenilejusticedomains.Indicatorsinthesetwodomainswereselectedfortheinitialrolloutofthenewplan,giventhattheyarecurrentlycollectedthroughexistingstatewidedatabases(TrailsandICON/Eclipse).AlloftheseindicatorswillbemeasuredamongyouthwhoparticipateinISSTs.Theselectedsetofindicatorsis:
Number(rate)withnewopeninvolvementsinTrailsafterCMPservicesbegan Number(rate)withnosubstantiatedabusefindingafterCMPservicesbegan Numberofmovesexperiencedwheninout‐of‐homeplacement Number(rate)dischargedtoapermanenthome Number(rate)whosuccessfullycompleteprobationand/orparole Number(rate)ofrevocationsbytechnicalviolationswherecaseresultedin
unsuccessfultermination Number(rate)whorecidivate,resultinginunsuccessfulcompletionofprobation
and/orparoleCMPsindicatewhichoftheabove‐listedoutcomesalignwitheachcase(i.e.,whichreflectinterventiongoalsorrepresentpotentialoutcomesrelevanttothechildbeingserved).ThisinformationwillbeusedtoidentifysubsetsofCMP‐servedchildrentoincludeintheanalysisofeachoutcome.Intheupcomingyear,inconjunctionwiththeSSCandEC,acollaborativeprocesswillbedevelopedtocreateastandardmeasurementapproachforindicatorsintheeducationandhealth/mentalhealthdomains.
iii. Proposedanalysis
OMNIevaluatorshaveestablishedprocesseswithresearchstaffinthestatechildwelfareandjudicialagenciestoconductaggregatedanalysesoftheselectedoutcomes.Usingthestateidentifiernumbersandclientidentifyinginformation,researchstaffattheseagencieswillmatchclientinformationwithoutcomedataextractedfromTrailsandICON/Eclipsesystems.Theywillthencalculateoutcomes,aggregatedatlocalandstatelevels,and
55
providetheseresultstoOMNI.OMNIwillthenanalyzetheprocessdatacollectedthroughETOorextractedfromexistinglocaldatabases.AnalysisofprocessdatawillresultinanaccountingofthetotalnumberofchildrenservedthroughISSTs,arrayofreferralsources,extentandpatternsofmulti‐systeminvolvement,familiesinvolvedinserviceplanning,andthetypesofoutcomesISSTservicesareattemptingtoachieve.OMNIwillsharerelevantresultswitheachlocalCMPandinthestatewideevaluationreport.Thestatewidemeasurementplanisonecomponentofamulti‐methodapproachtoevaluatingtheCMPinitiativeandreportingonmandatedlegislativegoals,asrequiredbythelegislation.Otherdatacollectionactivities,includingmid‐yearandend‐of‐yearreportingandadministrationofsurveys,willcontinueinordertoinformtheevaluationinotherkeyareasandtocomplywithotherlegislativerequirements.Withtheestablishmentofthestatewidedatamanagementsystemandclient‐trackingprocesses,thereistheopportunitymovingforwardfortheinitiativetoidentifyadditionalrelevantstandardoutcomestotrackovertime.OutcomesthataretargetsofstatewideeffortscouldbeassessedamongCMP‐servedyouthinordertoexploretheimpactofthecollaborativeapproach.Forexample,recommendationsinanOctober2011AnnieE.CaseyFoundationreportsuggestthattheColoradoDepartmentofHumanServicesfocuseffortsonspecificoutcomes,includinguseofcongregatecare,family‐basedplacements,andre‐abuserates.TheseoutcomesalignwithCMPtargetsandgoals,andcouldbeintegratedintothestatewidemeasurementplan.
56
YearthreeevaluationdirectionsInthefirstyearofevaluation,quantitativeandqualitativeresearchmethodswereemployedtogatherabaselineunderstandingofCMPcorecomponents,structures,andprocesses(i.e.,phaseIformativeevaluation).Theresultingfindingsinformedaseriesofrecommendationsthatwereinitiatedinyeartwo.Principally,thesecenteredaroundtwoareas:1)identifystandarddatacollectionopportunitiesacrossCMPsandestablishmeasurementsystemstosupporttheseefforts;and2)continuetoinformCMPbestpracticesandprogrammodelsthroughongoingprocessandformativeevaluationefforts.EvaluationactivitiesplannedforthethirdyearareintendedtoincreasecapacityandenhancedataqualityinwaysthatwillallowforanexaminationofCMPoutcomes(i.e.,phaseIIIsummativeevaluation).Theshiftawayfromformativeevaluationandinfrastructurebuildingtoasummativedesignwillrequirelocaladjustmentsindatacollectionpracticestoensureconsistentmeasurementstatewide.Thismaybechallengingtosomeprojects,andOMNI,CDHS,andtheEvaluationSubcommitteearecommittedtosupportingCMPsinmakingthesechanges.ThefollowinginformationoutlinesworkthatwilloccurinthethirdyearoftheevaluationtosupportmorerigorousmeasurementofCMPoutcomes.A. Continuetodevelopandimplementsystemstocollectandanalyzeuniform
measuresofCMPoutcomes,outputs,andprocesses1. Continueeffortstofocusoutcomemeasurementonasmallsetofwelldefined,uniformly
measuredoutcomesacrossCMPsonCMP‐targetedpopulations.Thenewstatewidemeasurementsystemimplementedforthe2011‐2012fiscalyearfacilitatestheanalysisofstandardoutcomesinthechildwelfareandjuvenilejusticedomains.ThesedataaremeasuredonchildrenandfamiliesservedbylocalISSTs,enablingustodrawadirectlinkbetweenprogressonoutcomesovertimeandCMPprograms.Inthe2011‐2012fiscalyear,mechanismstomeasurestandardoutcomesintheeducationandhealth/mentalhealthdomainwillbecollaborativelydeveloped,andimplementedinfiscalyear2012‐2013.
2. Continuetodevelopstandardoutputandprocessmeasuresandmeasurementapproachestosupportimproveddatacollectionandanalysisofprogressonlegislativegoals,atthestatewideandlocallevels.Thesetofoutputandprocessindicatorsmeasuredinthenewstatewidemeasurementsystemdescribedinthesectionaboveinformeffortstoassessreductionsinduplicationandfragmentationandtheextentoffamilyinvolvementin
57
serviceplanning.Inaddition,theevaluationwillcontinuetoassesscollaborativeeffectiveness,throughstakeholdersurveysandannualreporting.Intheupcomingyear,OMNIwillworkwithlocalCMPstofurtherrefineanddevelopadditionalmeasurementstrategiesintheseareas,aswellasimprovementsinquality,effectiveness,andappropriatenessofservices.Specifically,theon‐linestatewidemeasurementdatacollectionsystemprovidesastructureandmechanismbywhichlocalCMPscoulddevelopadditionaluniformmeasuresofCMPprocesses.Thesecouldincludethetrackingoffamilyengagementandsatisfactionwithservices,typesofinterventionservices,andotheraspectsofservicedelivery.
3. Developstandardmeasuresofcostandcost‐savingstobeimplementedacrossCMPs.
Formativeevaluationeffortshaverevealedsignificantdifficultiesinmeasuringlocalcosts,costreductions,andpotentialcostsavings.Utilizingtheinformationgathered,aswellasevidenceandexamplesfromthefield,OMNI’sevaluationteamwillworktoidentifycommonCMPcomponentsforwhichcostscanbeassigned,anddevelopasetofstandardizedmeasurestoassesscostsandcostreductionsassociatedwithCMPservicedelivery.MeasureswillberefinedincollaborationwiththeECandCMPcoordinatorsandthenimplementedacrossCMPsinfutureyears.
4. MonitorandtracktrendsinstatewidemeasuresofkeyCMPoutcomes.OMNI,inpartnershipwiththeEC,willdevelopanon‐lineplatformtodisplayrelevantdatainordertobringawarenesstoCMPinitiativeeffortsamongleadersandcommunitymembersthroughoutthestate.Thison‐linedatadisplaywillenablemonitoringoftrendsovertimeinthefourCMPoutcomedomains(childwelfare,juvenilejustice,education,andhealth/mentalhealth)atthelocal‐andstate‐level.
B. ContinuetodefineandsupportdisseminationofinformationoneffectiveIOG,
ISST,andcollaborativepractices1. SummarizeresearchonbestpracticesrelatedtocoreCMPcomponents.OMNIevaluators
areleadingeffortstoreviewresearchfindingsinthesystemsliteraturewhichwillbesharedwithlocalCMPsandstakeholdersintwopublicationstobedevelopedinthe2011‐2012fiscalyear.Theseincludethefollowing:
“SystemsFacilitationGuide.”Thisreportwillsummarizetheextantresearchliteratureonsystems‐levelcollaborationandthecorecomponentscontributingtoCMPstructuresandprocesses(e.g.,familyinvolvement,integratedserviceplanning).Thegoalistoidentifybestpractices.
58
Briefevaluationandresearchreportsonkeytopics.Intheupcomingyear,theevaluationteamwillcreatequarterlyevaluationnewslettersexploringtopicsrelevanttostateandlocalCMPstakeholders,suchasmeasurementofcostsavings.
2. SupporttheCMP’sbroadereffortstodefineanddisseminatebestpracticesinlocalCMPimplementationandoperations.OMNIstaffwillparticipateinanumberofeffortsrelatedtodisseminatingbestpractices.Theseincludethefollowing:
CMPHandbookdevelopment.OMNIwillcontributetoeffortsbeingledbyCMPcoordinatorstodevelopaCMPHandbook,whichwillincludekeyinformationtoorientnewCMPsandnewstakeholderstotheinitiative,theestablishinglegislation,SSCpoliciesandprocedures,localevaluationandmeasurementprocesses,andcollectivelearningsregardingeffectiveimplementationofIOGandISSTstructuresandprocesses.
CMPinformationsharingactivities.OMNIwillcontinuetosupportCMPcommunitynetworkingactivities,throughmaintainingtheCMPSharePointportal(anon‐lineinformationsharingtool)andattendingand/orfacilitatinglocally‐initiatedregionalmeetingsfocusedonCMPimplementationtopics.
Familyinvolvementtechnicalassistance.OMNIwillcontinuetosupporttheFamilyVoiceandChoiceCommittee’seffortstoexamineevaluationfindingstoidentifytechnicalassistanceneedsrelatedtofamilyinvolvementacrosstheinitiative.
Incentivefunding.OMNIwillfacilitatetheSSC’sdiscussionsregardingthedevelopmentofnewmethodstodetermineperformancebasedincentivefundingallocations.
Movingforward,theevaluationwillfocusoncontinueddevelopmentofmeasurementinfrastructuretocreateasolidfoundationforsummativeevaluationofCMPeffortsstatewide.OMNIwillcontinuetogroundtheevaluationinaparticipatoryapproach,workinginclosecollaborationwiththeCMPStateSteeringCommitteeandtheEvaluationSub‐committee,inanefforttoensurethatresultsarewellalignedwiththeneedsandinterestsoftheinitiativeanditsstakeholders.
59
Endnotes iColoradoRevisedStatute,Title24,Article1.9.(2010).Retrievedfromhttp://www.michie.com/colorado/lpext.dll?f=templates&fn=main‐h.htm&cp.iiCaliforniaDepartmentofEducation.(2007).Handbookondevelopingandevaluatinginteragencycollaborationinearlychildhoodspecialeducationprograms.RetrievedJune1st,2010fromhttp://www.cde.ca.gov/sp/se/fp/documents/eciacolbrtn.pdf.iiiU.S.DepartmentofHealthandHumanServices.(2010).Guidingprinciplesofsystemsofcare.RetrievedJune1st,2010fromhttp://www.childwelfare.gov/pubs/soc/socc.cfm.ivEmshoff,J.G.,Darnell,A.J.,Darnell,D.A.,Erickson,S.W.,Schneider,S.&Hudgins,R.(2007).Systemschangeasanoutcomeandaprocessintheworkofcommunitycollaborativesforhealth.AmericanJournalofCommunityPsychology,39,255‐267.vHicks,D.,Larson,C.,Nelson,C.,Olds,D.,&Johnston,E.(2008).Theinfluenceofcollaborationonprogramoutcomes:TheColoradoNurseFamilyPartnership.EvaluationReview,32,453‐477.viIbid.viiChrislip,D.D.,Larson,C.E.(1994).Collaborativeleadership:Howcitizensandcivicleaderscanmakeadifference.SanFrancisco:Jossey‐Bass.viiiBrunsEJ,SatherA,StambaughL.(2008).Nationaltrendsinimplementingwraparound:ResultsfromtheStateWraparoundSurvey,2007.In:Bruns,E.J.&Walker,J.S.,(Eds.),Resourceguidetowraparound.Portland,OR:NationalWraparoundInitiative,ResearchandTrainingCenterforFamilySupportandChildren’sMentalHealth.ixAnnieE.CaseyFoundation.(2002).Teamdecisionmaking:Involvingthefamilyandcommunityinchildwelfaredecisions.FamilytoFamilyToolsforRebuildingFosterCare.RetrievedNovember1st,2009fromhttp://www.aecf.org/upload/pdffiles/familytofamily/f2f_tdm_sept_02.pdf.xHenggeler,S.W.,Schoenwald,S.K.,Borduin,C.M.,Rowland,M.D.,&Cunningham,P.B.(2009).Multi‐systemictherapyforantisocialbehaviorinchildrenandadolescents.NewYork:Guildford.xiAlexander,J.,Pugh,C.,Parsons,B.,&Sexton,T.(2000).FunctionalFamilyTherapy.InD.S.Elliott(Ed.),BlueprintsforViolencePrevention(Vol.3).Boulder,CO:VenturePublishing.xiiBotvin,G.J.,Griffin,K.W.,Paul,E.,&Macaulay,A.P.(2003).PreventingtobaccoandalcoholuseamongelementaryschoolstudentsthroughLifeSkillsTraining.JournalofChildandAdolescentSubstanceAbuse,12,1‐18.xiiiWebster‐Stratton,C.&Reid,J.(2010).TheIncredibleYears:Evidence‐basedparentingandchildprogramsforfamiliesinvolvedinthechildwelfaresystem.NewYork:JohnWiley&Sons.
60
xivAltman,J.C.(2005).Engagementinchildren,youth,andfamilyservices:Currentresearchandbestpractices.InG.Mallon&P.Hess(Eds.),ChildWelfareforthetwenty‐firstcentury:Ahandbookofpractices,policiesandprograms(pp.72‐86).NewYork:Columbia.xvGoerge,R.M.,Smithgall,C.,Seshadri,R.,&Ballard,P.(2010).IllinoisFamiliesandTheirUseofMultipleServiceSystems.Chicago:ChapinHallattheUniversityofChicago.
62
AppendixA.StructureofstatemanagementoftheCMP
StateSteeringCommitteeProvidesprogramoversight,advancessystemreformgoalsofCMPlegislation,composedofrepresentativesoforganizationsofmandatorysignatories,familyadvocacy,andparticipatingcounties.
EvaluationSubcommitteeGuidesstatewideevaluation,promoteseffectiveevaluationpracticesatthecountylevel,andconsultswithOMNIInstitute
Communications/MarketingSubcommitteeGathersanddisseminatesinformationaboutcollaborativeactivitiestostateandlocalstakeholders,facilitatesinformationsharing
FamilyVoiceandChoiceSubcommitteeCo‐ledbyalocalCMPcoordinatorandstatefamily‐drivenorganization,thiscommitteegathersandsharesinformationaboutpromisingstrategiestoengagefamiliesattheIOG,ISST,andservice‐deliverylevels.
SustainabilitySubcommitteeSupportssustainabilitythroughpoliticaladvocacy,strategicplanning,grantwriting,andcultivationoforganizationalpartnerships.
DHSProgramDirectorProvidesguidanceandoversight,reviewscounty‐levelmemorandaofunderstandingandannualreports,allocatesfundsandincentivefunds
63
AppendixB:Additionalreportsandproducts
BelowaredescriptionsofselectadditionalreportsandproductsdevelopedduringthecourseoftheCMPstatewideevaluationtodate.ThesedocumentscanbeaccessedusingthefollowinglinktotheCMPinitiativeportal(notethatrecentreportsmaystillbeunderfinalreviewandthusmaynotbepublishedonthissiteyet):http://collaboration.omni.org/sites/1451/SitePages/Home.aspxReportspublishedin2010
Report Date Description
CollaborativeManagementProgram:Year1EvaluationProgressReport
February2010
ThisreportdescribesOMNIInstitute’squalitativeandquantitativeapproachtostatewideevaluationforthefirstyearofevaluation.Thereportdiscussesinitialformativefindings,addressingtopicssuchasbarrierstoCMPimplementation,resourceallocationconflicts,IOGperformanceissues,andobservedsuccessoflocalCMPimplementation.
StateManagementSurveyFindings November2010
InFall2010,atotalof29StateSteeringCommitteemembersandIOGCoordinatorsrespondedtotheStateManagementSurvey,anonlinesurveydesignedtoelicitfeedbackregardingCMPmanagement,progress,andareasforfurtherdevelopment.ThisreportpresentstheresultsoftheStateManagementSurvey,whichincludesitemsthatassessprogressandfunctioningoftheStateManagementOffice,theStateSteeringCommittee,theFamilyVoice&ChoiceSubcommittee,theEvaluationSubcommittee,andtheoverallHB1451initiative.
CollaborationandOverallSuccessSurvey:InitialDataSummary
December2010
ThisreportprovidesasummaryofinitialfindingsfromtwosurveysadministeredtoIOGmembersin22CMPsinSpring2010.ThesurveysassessedIOGmembers’perceptionsoftheirCMP’scollaborativeprocesses,aswellastheperceivedsuccessoftheircollaborativeeffortsonseveralgoalsofthestatelegislation.Thefindingsprovideaninitialassessmentofcollaborativeprocessesandoutcomes,animportantpartofdemonstratingthebenefitoftheCMPapproach.
SummaryofSelectChildWelfareDataIndicatorsbyCounty
June2010
Thisreportpresentscounty‐level“snapshots”ofperformanceonsixchildwelfaredataindicatorsforthetimeperiodof2003to2009(representedinaveragesacrossyears),asdefinedbytheU.S.DepartmentofHealthandHumanServicesandreportedtotheColoradoDepartmentofHumanServicesthroughtheTrailsdatabase.Thisreportalsoprovidesgeneralbackgroundinformationontheavailablecounty‐levelstatewidedatarelatedtoCMPefforts.
64
Report Date Description
EmergingEvidenceSupportingCollaborativeManagementPrograms
July2010
OMNIInstitutepreparedabriefdocumentinSummer2010detailingevidencethatsupportsthesuccessofcommunitycollaborativestomeetlegislationgoals.ThedocumentincludesasummaryoftheSystemsofCare(SOC)philosophyandhighlightstheHighFidelityWraparoundapproachasonepossiblewaytoimplementcollaborativecarepracticestoimprovechildandfamilyoutcomes.
CollaborativeProcessesandEmergingBestPractices
August2010
Thisreportsummarizestheresultsofin‐depthinterviewsconductedwithasampleofCMPstakeholders(localIOGmembers)andresponsestoopen‐endedquestionsfromasurveyonthequalitiesofcollaborationefforts.Thereportprovidesadiscussionofthemajorthemesthatreflectfindings,alongwithabriefpresentationofpotentialbestpractices.
CollaborativeManagementProgram:Year1StatewideEvaluationFindings
October2010
Thisreportsummarizesfindingsandlearningsemergingfromevaluationeffortsoverthefirstyearofthestatewideevaluation.ThereportdetailsOMNIInstitute’sworktodateonidentifyingcommonanduniquecomponentsofCMPserviceandsystemsdeliverymodelsandimplementation;measuringthequalityofcollaborativestructures,relationships,andprocessesaswellastheperceivedeffectivenessofthecollaborativeapproach;buildinginfrastructureandprocessestoassessprogressonlegislativegoalsinastandardwayacrossCMPs;identifyinghowCMPsaremeasuringlocalprogramimpactsbyidentifyingeachCMP’stargetedpopulationsandchild/familyoutcomes,dataindicators,andperformancegoals;andassessingthecapacitytoassesscross‐CMPimpactsonkeypopulation‐leveloutcomesandprovidingrecommendationsandguidanceinestablishinginfrastructuretomovetowardsstatewideoutcomeanalyses.
65
Reportspublishedin2011
Report Date Description
CollaborativeManagementProgram:Year2EvaluationProgressReport
February2011
BuildinguponfindingsfromYear1,statewideevaluationeffortsinYear2involvedfocusingmeasurementonasmallsetofwell‐defined,uniformlymeasuredoutcomes.Thisreportsummarizestheresultsandrecommendationsthatledtothisfocus,resultinginthedevelopmentofanOutcomesWorkgroup;describestheOutcomesWorkgroupprocessandprogress;presentsthemeasurementplanthatemergedfromthisprocess;anddetailstheproposednextstepsintheprocess.
StateMeasurementPlan:BriefSummaryforCMPCoordinatorsandIOGs
March2011
OMNIdevelopedabriefsummaryofthestatemeasurementplan(seepreviousentry)fordisseminationamongCMPcoordinatorsandfortheirIOGs.Thepurposeofthissummarywastodetailtherationale,objectives,dataindicators,datacollectionstrategies,andnextstepsfortheimplementationofthestatemeasurementplan.ThisdocumentwasintendedtopresentkeycontentabouttheOutcomesWorkgroup’seffortsandtheresultingstatemeasurementplaninaformatappropriateforwidedistribution.
StateMeasurementPlan:PowerPointPresentationforCoordinators
April2011
Inadditiontothebriefsummaryjustdescribed,thispresentationwasdevelopedasatooltoassistCMPcoordinatorsintheireffortstodescribethestatemeasurementplantoIOGmembersanddiscusslocalCMProlesandresponsibilities.Likethetwoprecedingresources,thepresentationprovidesasummaryoftheOutcomesWorkgroup’sefforts,thestatemeasurementplanthatemergedfromtheWorkgroup’splanningprocess,andnextstepsforimplementingthestatemeasurementplan.
CurrentPracticesinthePreventionofChildAbuseandNeglect
April2011
Thisbriefreportprovidesanoverviewofevidence‐basedprogramsandinterventionsmostwidelyrecognizedintheareaofchildabuseprevention.Alsoincludedarediscussionsaboutthetheoreticalframeworksthatprovidethebasisfortheseevidence‐basedpractices,aswellaslinkstomoreinformationaboutadditionalpromisingprograms.
CollaborationandOverallSuccessSurvey:SummaryTable
June2011
ThisbriefdocumentprovidesasummarytableofaggregateCollaborationandOverallSuccesssurveyresultsfromsurveyre‐administrationinSpring2011(seeCollaborationandOverallSuccessSurvey:InitialDataSummary–December2010abovefordetailsregardingthesurvey).Dataresultsforanumberofsurveydomainswerereportedbyfiscalyear,bycoordinatorstatus,andbypopulationdensitycharacteristics.
66
Report Date Description
CollaborationandOverallSuccessSurvey:County‐LevelSurveyResults
July2011Thesereportsprovidedescriptiveinformationregardingeachcounty’sIOGmembers’generalperceptionsoftheirCMPstructure,collaborativeprocesses,andemergingoutcomes.TheresultsinthesereportsreflectCMPsscoresfortwofiscalyears(2009‐10and2010‐11).
FamilyInvolvementSurveyReport August2011
InSummer2011,theFamilyVoiceandChoiceCommitteecollecteddatarelatedtoCMPfamilyinvolvementinordertounderstandfamilyadvocacyactivityasimplementedamongCMPcounties.Thisreporthighlightspreliminaryfindingsfromdatacollectionefforts,includinganoverviewofcurrentfamilyadvocacyactivities,aswellasadiscussionforresourcedevelopmentandtrainingopportunitiesforCMPcounties.
67
AppendixC.SourcesofadditionalfundingreportedbyCMPs
FundingType
SourceofFunds DescriptionNumberofCMPsreporting
TotalAmountsReceived
Federal
JuvenileAccountabilityBlockGrant
Fundsprovidedasblockgrantstostatesforprogramspromotinggreateraccountabilityinthejuvenilejusticesystem
1 unreported
BureauofJuvenileAssistance
Fundsprovidedforprogramsaimingtoimprovecriminaljusticesystemsandservicesforcrimevictims
1 unreported
SupportSystemsforRuralHomelessYouth
FundsawardedtoColoradotosupportprogrammingforhomelessyouthandfamilies
1 $15,000
BasicCenterProgram
GrantfromFamilyandYouthServicesBureautosupportorganizationsandsheltersthatserveandprotectrunaways,homeless,andstreetyouth
1 $15,000
PromotingSafeandStableFamilies
FundsdistributedbyUSHSStostatestograntservicesthataddressfamilysupport,familypreservation,reunification,andadoption
2 $73,500
OfficeofJuvenileJusticeandDelinquencyPrevention
Fundsforjuvenilejusticeprogramming
1 unreported
State
DivisionofCriminalJusticeJuvenileAssistanceGrants(JAG)
Competitivefundingfordevelopmentofprogramsthatpreventorreducecrimeanddelinquencyusingcollaborativeevidence‐basedandpromisingpractices
6 $415,000*
TonyGrampsasYouthServicesGrant
Competitivefundingtolocalorganizationsthattargetyouthandtheirfamilieswithprogramsdesignedtoreduceyouthcrimeandviolence
1 unreported
StatewideStrategicUseFund
FundsdistributedforprogramsservingColorado'smostvulnerablefamilies
2 $382,489
DivisionofBehavioralHealthPreventionBlockGrant
Fundsforpreventionprogramming 1 $98,787
Privatefoundation/
other
UnitedWay Fundsforsocialservices 1 $81,000
CenterforJuvenileJusticeReformTechnicalAssistanceGrant
Providestechnicalassistanceforimplementinganevidence‐basedcollaborativejuvenilejusticepracticemodel
1 unreported
ElPomarFoundationBuildaGenerationGrant
Fundsforsocialprograms 2 $4,000
68
AppendixD.FrequentlyuseddatasourcesformeasurementofCMPperformancegoals
Domain DataSource Description
ChildWelfare
Trails
Trailsisastatewideautomatedcasemanagementsystemdetailingclientinformationacrossyouthcorrectionsandchildwelfarepopulations.TrailsiscurrentlyusedbyvariousdivisionsoftheColoradoDepartmentofHumanServices.
JuvenileJustice
Eclipse/ICON
ICON/Eclipseisastatewideautomatedcasemanagementsystemforthedistrictandcountycourts,andisusedbytrialcourtsandprobation.ICON/EclipselinkstotheColoradoIntegratedCriminalJusticeInformationSystem(CICJIS)whichisusedbytheDivisionofYouthCorrectionsandotherlawenforcementagenciestotrackoffenses.
Trails Seeabove
Education
Localschooldistrictrecords
Districtsmaintainlocaldatabasesthattrackabsences,tardiness,andotherstudent‐leveldata
Web‐basedrecordsmanagementprograms(PROSTAR,GoEdu.comandInfiniteCampus)
Onlinedatatrackingsystemstorecordattendance,grades,studentinformation;withschool,teacher,parent,andyouthlevelaccess
NWEADistrictTesting Standardizedmeasuresofacademicprogress(MAP)testsimplementedindistrict‐widetesting
Health/MentalHealth
CCAR/DMH
CDHS’sColoradoClientAssessmentRecords(CCAR)isamulti‐dimensionalchecklistandleveloffunctioningmeasurethatisconsistentlycollectedatintakeanddischargeinseveralmentalhealthsettingsacrossthestate,includinginpatientandsubstanceuseprograms.
DACODS/DBH
DACODSisaclientleveldatacollectioninstrumentusedbytheDivisionofBehavioralHealth(DBH)totracksubstanceusetreatmentandprogramimplementationvariables
CHIPS/CMHCprogramCentennialMentalHealthCenter’sinformationsystemtrackingclient‐levelmentalhealthservicerelateddata