Measuring Our Success

Kristen Sanderson, MPH, CHESProgram Coordinator, Safe Kids Georgia

1

Objectives

• Describe our quality improvement process• Defining metrics and what to measure• Improve data collection process• Challenges and successes

• Discuss program evaluation strategies• Describe our assessment process• Next steps

2

Our Objective

• Define a set of meaningful metrics that reflect our progress in reducing incidence and severity of child injuries and mortality in the six focus areas.

3

Challenges

• Measuring behavior change

• Many factors that influence safety and the incidence of injury. i.e. laws, law enforcement, community development, economics, cultural norms

• Measuring not just one program, but many programs and overall effectiveness

4

QUALITY IMPROVEMENT AND PROCESS EVALUATION

5

Monitoring vs. Evaluation

• Before we could begin measuring our impact (evaluation), we needed to collect more accurate data on our activities and programs (monitoring)

• Monitoring • Monitor the implementation process• On-going measurement of performance• Regular tracking of resources, activities, and outputs

• Are things working well?

• Evaluation• Assess specific program outcomes and impact

• What is the result?• E.g. behavior change

6

First Step

• Analyzed current method of coalition reporting• Pros:

• Collecting program activity information including # of events by focus area, # of materials/equipment distributed, people reached

• High response rate

• Cons:• Annual reporting = inaccurate, untimely data• Some questions unclear = different interpretations and

unreliable information

• Current methods were not accurate, timely, or effective

7

Our Tasks

1. Improve Quality of Program Data• Establish metrics• Revise reporting forms• Improve data collection process

2. Assess Our Activity and Impact• Quarterly and Annual reports – monitoring • Periodically assess program impact and identify areas

for improvement

8

Establishing our Metrics• Determine what data we want to and can collect

• Questions we asked ourselves: • What do we want to measure? • What data do we want to track over time?• What can we feasibly collect from our coordinators?• What information will be useful for the coordinators?

• What did we decide to measure/track?• Injury statistics by county• County demographics• Zip code (location) of events• Event details

9

Improving our Data Collection Process

• Goal 1) Collect data as close to time of event or program activity as possible Improve accuracy and reliability of the reported data Provide more timely feedback for program adjustments

• Goal 2) Make data collection as easy and fast as possible Reduce the # of non-reporting coalitions Recognize the diversity of coalition members’ roles (part-

time vs. full-time, e.g.)

• Solution: Annual reporting Quarterly Reporting

10

Quarterly Report

11

Quarterly Report

12

Quarterly Report - Inventory

13

Challenges

• Change• History of no repercussions for not reporting

• Estimating numbers

• No incentives or disincentives

• Varying skill levels and knowledge of Excel

• Buy-in• Benefits of reporting never communicated

• Time intensive

14

Overcoming Challenges & Getting Buy-In

• Discussions at meetings, one-on-one site visits• Technical assistance and feedback• Follow-up: reminders, phone calls• “What do I get out of it?”

• Accurate numbers for lead agency, funders, sponsors• Quarterly/Annual reports showcasing numbers• Not having to remember all activity at the end of the

year• Needed to keep 501(c)3 status

• User-friendly report

15

Quarterly Report

• Pilot testing • Full dissemination in 2nd quarter 2012• Lots of feedback, many revisions!

• Group revisions• Only released updated versions quarterly

• Final changes for 2013 • Process took over a year

16

Roadblocks

• Coalitions had multiple reports• CPAT

• Annual Survey

• Funder requests

• SKW Grants

• Technical difficulties• Many issues with the new form required lots of revisions

• Balancing multiple stakeholders’ expectations• Board committees• Lead agencies• Coordinators

17

Early Successes• Increases in number of reporting with each

quarter• More accurate and timely data• Standardized process

18

Recommendations

• Involve coordinators and all other stakeholders throughout the process• What does everyone want to get out of this? What are

the benefits for everyone? Challenges?• Talk with coordinators who report and don’t report – why

aren’t they reporting?

• Provide training or meeting before dissemination of new tool• Training/explanation beforehand will minimize

inaccuracies and misinterpretation

19

Recommendations

• Obtain buy-in from coordinators early on• Talk with them one-on-one on the benefits for them, how

to help them• Know what other reports they already have

• 4. Pilot test and revise

• 5. Slow process – patience is needed!

20

MEASURING OUTCOMES AND ASSESSING OUR IMPACT

21

Program Evaluation - Initial Focus on 2 Areas

• Child Passenger Safety• Largest program• Has the most evidence supporting risk mitigation

approaches• Questions to answer:

• Do we need to do more, and how much more? (un-served population)

• Can we show reasonable impact between SK vs. non-SK coalitions?

• Behavior changes?• Poisoning

• Recently trended upwards in incidence• Least evaluated area, least implemented program• Question to answer:

• What needs to be done? What works?

22

Poisoning Prevention

• Program Development• Develop program based on literature review, existing evidence-

based programs, coordinator interviews• Develop capacity • Provide resources to coordinators• Create sustainability

• Evaluation • Evaluation of instructor training• # of trained Poison Prevention Instructors (capacity)• # of Poison Prevention educational events (increase in education)• Pre- and post-test forms (change in knowledge, behavioral

intentions)• 3 month follow-up (behavior change)

23

Child Passenger Safety

• # of CPSTs in coalition counties (capacity)

• Training (capability)• # of certification classes• # of recertification classes

• Car seat checklist forms• Monitoring misuse overtime

24

Child Passenger Safety

• Follow-up evaluation Pilot Project• 3 month follow-up after car seat checks• Demonstrate knowledge retained and behavior change• Partnership with Georgia Department of Public Health

• Impact of Booster Legislation (July 2011)

• Challenges• Lots of data from multiple sources• Narrowing information• Link between injury data and our activities

25

Assessment

26

Injury and Death Statistics•For all 6 of the focus areas•Compare to prior 2-3 years

Program Statistics•Activity levels by program, by coalition•Location of activities•Coalition building activities and interaction with other agencies•Program funding & grants•Staff resources devoted per program

Program Evaluation & Recommendations•Program specific outcome evaluation•Summary of how SKG “moved the needle”•Narrative success stories and lessons learned•Coalition activity and assessment of effectiveness (paid v. volunteer)•Changes to programs and coalition building•Additional resources needed (staff and $)

Next Steps

• Monitor program activity• Continue to improve # of coalitions reporting• Create database for Program Activity Data• Create and distribute quarterly and annual reports

• Develop goals and objectives• Track our progress

• Identify injury prevention program priorities

• Outcome evaluation• Program-specific evaluation

27

28