cfc measurement presentation€¦ · cfc measurement presentation author: kathryn goldsworthy...
TRANSCRIPT
![Page 1: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/1.jpg)
MeasuringOutcomes
Dr.CatherineWadePrincipalResearchSpecialistParentingResearchCentre
SamHaskardManagerChildrenandFamiliesProgram
AnglicareSA
![Page 2: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/2.jpg)
Overview
Catherine:
• Approachestomeasuringprogramoutcomes
• Howtoeffectivelysourceandselecttherightmeasures
ü Interpretingpsychometrics
ü Measuresdatabases
ü Useofvalidatedtoolsvstailoredmeasures
Sam:
• Apracticalexample
Catherine&Sam:
• Exploringoptions
![Page 3: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/3.jpg)
Definingprogramoutcomes
• Gettingclarityonoutcomes– importanceofprogramlogicinclearlyarticulatingthelinkbetweenprogramactivitiesandanticipatedoutcomes.
![Page 4: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/4.jpg)
Whatoutcomestomeasure?
![Page 5: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/5.jpg)
Whatoutcomestomeasure?
Trytoincludemeasuresforboththekeymediatingandintendedoutcomes,however:
• Considertimeframefortheinterventionandevaluation– whichintendedoutcomescouldwereasonablyexpecttoseeoccur?Someintendedoutcomesmightbebeyondthescopeoftheevaluation.
• Considerfeasibilityofmeasurementi.e.,willitbepossibletoobtainameasureofthetargetedintendedoutcome/s?
![Page 6: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/6.jpg)
Whatoutcomestomeasure?
• Consideralsoincludingmeasuresofkeyimplementationandengagementoutcomes:
Implementationoutcomes considerwhatcriticalquantity,qualityandtimingoutcomesoftheprogramimplementationarenecessarytoachieverequiredengagementandadherencewithprogramsstrategies(e.g.,measuresofimplementationfidelity)
Engagementoutcomes considerwhatlevelsofengagementandadherencewithprogramstrategiesarethoughttobeessentialtoachievemediatingoutcomes (e.g.,measuresofclientattendanceandengagementwithprogram)
![Page 7: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/7.jpg)
Whatoutcomestomeasure?
![Page 8: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/8.jpg)
Approachestomeasuringoutcomes
• Quantitativeversusqualitative– wherepossiblecanbebeneficialtoincludeboth.
• Considerneedsoftheevaluationandthetargetparticipantgroup:
• Rigourofapproach
• Culturalappropriatenessofmethod(e.g.,useofyarningcirclesforevaluationsinvolvingAboriginalfamilies)
• Feasibilityofapproachandacceptabilityofmethodsbyfamiliesandprogramfacilitators
![Page 9: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/9.jpg)
Identifyingandselectingmeasures
• Identifyatleastoneindicatorforeachoutcomeofinterest– engagement,implementation,mediatingandintendedoutcomes
• Astrongindicatorwillprovideameasureofthatoutcomethatisclosesttoa‘realworldresult’– morereliableandvalid
• Aweakindicatorwillbeanapproximationonlyandinvolvegreatererror– lessreliableandvalid
![Page 10: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/10.jpg)
Identifyingandselectingmeasures
![Page 11: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/11.jpg)
Identifyingandselectingmeasures
![Page 12: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/12.jpg)
Typesofmeasures
Observational(video/live/audio):providedirectmeasuresoftheparentand/orthechild'sbehaviour,parent-childinteractions,skillacquisition,andgeneralisationofskillsoutsideoftheprogram.Checklists&toolsavailable,ordevelopyourown.
Self-monitoring:e.g.parentrecordsfrequencyordurationofanevent/behaviour,ortopographyofbehaviour(A’s,B’sandC’s).
Self-report:Checklists,ratingscales,questionnaires.
Interviews:structured/pre-preparedscheduleofquestions&prompts.Acquiescence&socialdesirabilitycanbeanissue.
Directtesting:knowledgeorskillassessmenttools(e.g.,vignettes,quizzeschilddevelopmentalassessments).
![Page 13: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/13.jpg)
Measuresdatabases
CfC outcomesmeasurementmatrixonCFCAwebsite:https://aifs.gov.au/cfca/expert-panel-project/families-and-children-outcomes-measurement-matrix
OntarioCentreofExcellenceforChildandYouthMentalHealth: http://www.excellenceforchildandyouth.ca/resource-hub/measures-list-search-results?p=all
UniversityofMinnesotaEvaluationInstrumentSearch:https://cyfar.org/search_evaluation_instruments
MeasurementInstrumentDatabasefortheSocialSciences:http://www.midss.org/tools/search
TheCaliforniaEvidence-BasedClearinghouseMeasurementToolsforChildWelfare:http://www.cebc4cw.org/assessment-tools/
![Page 14: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/14.jpg)
Usingvalidatedmeasuresversuscustomisedtools:
Prosofestablishedmeasures:
• Reliable
• Valid
• Normed
• Guidelinesforscoringandinterpretation
• Enablesbenchmarking
• Manyfreelyavailabletools
![Page 15: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/15.jpg)
Usingvalidatedmeasuresversuscustomisedtools:Howeverneedtoconsiderfeasibilityofusingestablishedtools:
•Fitforpurpose
•Version
•Restrictions
•Time,cost,length,reportingburden
•Instructions
•Skillsarerequiredoftherespondent
•Populationnormsavailable
•Usedwithclientslikeyours?
![Page 16: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/16.jpg)
Usingvalidatedmeasuresversuscustomisedtools:
Someimportantconsiderationsforthedevelopmentofcustomisedtools:
• Ensuresamequestionsareaskedatpreandposttest– nowordingchanges(thiscanleadtochangesinmeaningandmaythereforenotbemeasuringthesamethingpreandpost)
• Useclear,plainlanguageandavoiddouble-barrelledquestions
• Ensuregoodalignmentbetweentargetedprogramoutcomeandwordingofquestions.
![Page 17: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/17.jpg)
Usingvalidatedmeasuresversuscustomisedtools:
Someimportantconsiderationsforthedevelopmentofcustomisedtools:
• Considerwhetherpossibletoco-designquestionswithclientgroupandincorporateapilottestbeforestartingyourevaluationtoobtainclientfeedback
• Acknowledgelimitationsofusingnon-validatedmeasureswheninterpretingfindings.
![Page 18: CFC Measurement presentation€¦ · CFC Measurement presentation Author: Kathryn Goldsworthy Created Date: 11/1/2017 2:19:12 AM](https://reader034.vdocuments.mx/reader034/viewer/2022050109/5f46b8be5161704d3b4efe61/html5/thumbnails/18.jpg)
Dr.CatherineWadeLevel5,232VictoriaParadeEastMelbourne,Victoria,Australia3002
P:+61386603500F:+61386603599
@WadeCath