MIECHV Benchmark Data Collection and Evaluation

Mary Anne Wilson, M.S.W.Center for Prevention Research and DevelopmentInstitute of Government and Public AffairsUniversity of Illinois

Overview

• What are the benchmarks and why are they necessary?

• Federal requirements• Evaluation and data collection• Performance measures• Continuous quality improvement (CQI)

Purpose of Presentation Increase understanding for role of Benchmarks,

Benchmark Reporting Requirements Understand the reporting and data collection

requirements for the MIECHV programs Identify the roles that each organization involved

in Benchmark reporting will play, and what additional roles that CPRD can play for MIECHV and home visiting services

Identify key performance measures for MIECHV programs, both for Benchmarks and continuous quality improvement efforts

Understand the role of the field data collectors

And what is not the purpose of this presentation?

What is MIECHV?Maternal, Infant, Early Childhood Home Visiting

Over time, families and home visitors build partnerships and work together to:• Improve health and development.• Prevent child injuries, child abuse, neglect, or maltreatment,

and reduce emergency department visits.• Improve school readiness and achievement.• Reduce crime, including domestic violence.• Improve family economic self-sufficiency.• Improve the coordination and referrals for other community

resources and supports.

How do we define success?

• Providing high-quality home visiting services that positively impact the lives of children and families enrolled in the MIECHV programs

• Meeting and exceeding Benchmarks• Findings ways to pool our knowledge collectively to improve

statewide delivery of home visiting services.

How does MIECHV fit in?• MIECHV is designed to expand HV services into priority

populations:• Reside in communities in need of services & have low income• Include pregnant women who have not attained the age of 21• Have a history of child abuse• Have a history of substance abuse• Have users of tobacco products• Have a history of, or have children with, low student achievement• Have children with developmental delays or disabilities• Include members of the military

MIECHV Priority Communities

1. Rockford- Winnebago County

• 65.1% White, 20.5% African American, 15.8% Hispanic2. Vermilion County

• 83.6% White, 13.3% African American, 4.4% Hispanic3. Chicago- Cook County

• 45% White, 32.9% African American, 28.9% Hispanic4. Elgin City ( border of Cook and Kane Counties)

• 69.5% White, 7.4% African American, 43.6% Hispanic5. Cicero Town- Chicago

• 51.9% White, 3.8% African American, 86.6% Hispanic6. Macon County

• 79.8% White, 16.6% African American, 2.0% Hispanic

Doula Sites: N. Lawndale, E. Garfield Park, Vermilion County, Rock Island County and Waukegan

The program goals, per the HRSA website:

1. Strengthen and improve the programs and activities

2. Improve coordination of services for at-risk communities

3. Identify and provide comprehensive services to improve outcomes for families who reside in at-risk communities

MIECHV Benchmarks

• There are a total of 6 Benchmark areas

• These reflect overall domains in which we expect home visiting services to have an impact.

• Each benchmark has a set of constructs, and these total 37 constructs across the 6 benchmarks

• Benchmarks were developed in the Legislation, and the constructs were developed by HRSA to address the benchmarks.

Why do we have Benchmarks?

The legislation funding MIECHV required quantifiable, measurable improvements for the populations receiving services. Programs must demonstrate improvement in the following Benchmark areas:

1. Improved maternal and newborn health2. Prevention of child injuries, child abuse, neglect, or

maltreatment, and reduction of emergency department visits

3. Improvement in school readiness and achievement4. Reduction in crime or domestic violence5. Improvements in family economic self-sufficiency6. Improvements in the coordination and referrals for other

community resources and supports

Where do the Benchmarks come from?

Now, let’s look at the Benchmarks individually

• The Benchmarks contain constructs, specific measures to address each concept

• Reflect outcomes of home visiting services

Constructs: • Prenatal Care• Prenatal use of alcohol, tobacco, or illicit

substances• Postpartum use of contraception and

Interpartum Interval• Screen for Maternal Depressive Symptoms• Duration of breastfeeding, well-child visits,

and maternal health insurance coverage

1. Improved Maternal and Newborn Health

2. Prevention of Child Injuries, Child Abuse or Maltreatment and Reduction of Emergency

Department Visits

Constructs:• Visits for child and mother to the emergency

department from all causes• Information provided or training of participants on

prevention of child injuries• Incidence of child injuries requiring medical treatment• Reports of suspected maltreatment for children in the

program, reported substantiated maltreatment, and first time victims of maltreatment

3. Improvements in School Readiness and Achievement

Constructs:• Parent support and knowledge for children’s learning and

development• Parenting behaviors and parent-child relationship• Parent emotional well-being or parenting stress• Child’s communication, language and emergent literacy,

cognitive skills• Child’s positive approaches to learning and attention• Child’s social behavior, emotional regulation and emotional

well-being• Child’s physical health and development

4. Domestic Violence

Constructs:• Screening for domestic violence• Of families identified for the presence of domestic

violence, number of referrals made to relevant domestic violence services

5. Family Economic Self-Sufficiency

Constructs:1. Household income and benefits2. Employment or education of adult (mother and

father) members of the household3. Health Insurance Status

6. Coordination and Referrals for Other Community Resources and Supports

Constructs:1. Number of families identified and referred to available

community resources2. The number of families who complete referrals to available

community resources3. Number of agencies with which the home visiting provider has

a clear point of contact in the collaborating community agency 4. Number of agencies with which the home visiting provider has

established a formal memorandum of understanding

What the Evaluation is and is not• The Evaluation is:

• A way to measure and address the challenges in scale-up implementation of HV services

• A way to show Congress that tax dollars spent on additional HV services is important and useful

• A way to introduce statewide continuous quality improvement efforts

• A goal setting mechanism• The Evaluation is not:

• A performance assessment to be used for funding decisions• A method to weed out the poor performing HV sites• A way to collapse the variety of model HV programs into a single

government-managed HV organization

Illinois MIECHV: A Partnership Approach

CPRD is involved in the following areas:• In partnership with the state, developing measurement

procedures and systems which adequately address the Benchmark reports

• Coordinating Benchmark data collection with state and sites • Work together with home visitors to gather benchmark data

• Query data from MIECHV data system and prepare reporting to the feds annually

• Utilize data to track individuals through time in services• Collect home visiting services satisfaction information

• Help clarify possible barriers to engagement• Lead Home Visiting Continuous Quality Improvement efforts in

MIECHV and throughout the Home Visiting State System

What if we don’t make improvements on the benchmarks after 3 years?

• Would need to develop a plan of action

• CPRD, the Governor’s Office, and individual community

programs would work together to develop a plan and

next steps

• The CQI process is crucial to next steps

What is CQI and how will CPRD aid in the CQI efforts?

I can tell you with 95% confidence that there is less than a 65.6%

possibility that the CQI will simply generate 34.8% more meaningless

statistics.

What are the cohorts mentioned in the Benchmarks?

• The term “cohort” refers to the idea of a group of families

who enter services together.

• But families are entering and exiting HV services all the time

• Cohorts in MIECHV follow one group of people and compare them

to another group enrolled a year later

• Cohorts help us measure program change and effectiveness over

time

• How will we know if individuals are changing?

• Individual change is reviewed in CQI process

Who are Field Data Collectors?

• What does it mean to be a field data collector?

• How will we all stay in contact?

• How will we ensure regular communication and updates?

• Working with the home visiting sites and home visitor

• Will use a shared calendar through Google Calendar (E.g. 'macondc@gmail.com‘), e-mail, call and text.

The Role of Field Data Collectors in the Benchmarks

• Some Measures require rigorous approach for reliability

• The Benchmarks created a large and taxing load on home visitors

• Some benchmarks require that the parent not be hindered by watching their child, such as needing to fill out a longer assessment

On-site Data Collection Field data collectors will work with home visitors to collect

measures from families Assess the families at three time periods

2-4 weeks after enrollment1 year and 2 year follow-ups

Informed Consent – depending on age or emancipation status Parental consent – if mother is less than 18 years old, written

parental/guardian consent is required and self-assent Self-consent – emancipated or 18 years or older, written assent

is required Gift cards are provided to participants

What is the PICCOLO?Parenting Interactions with Children: Checklist

of Observations Linked to Outcomes

• PICCOLO items describe positive parenting

• PICCOLO helps practitioners identify parenting strengths to encourage parents to do more

What does PICCOLO measure?

From the research literature, there are identified four important kinds of “developmental parenting” that support child development:

• Affection• Responsiveness• Encouragement• Teaching

What does PICCOLO look like?

29 THINGS PARENTS DO THAT PREDICT SCHOOL READINESS

• 1. Speak warmly 2. Smile at child 3. Praise child 4. Stay physically close to child 5. Say positive things to child 6. Interact in positive ways with child 7. Show emotional warmth 8. Pay attention to what child is doing 9. Change activities to meet child’s interests or

needs 10. Be flexible when child changes interests 11. Follow what child is trying to do 12. Respond to child’s emotions 13. Look at child when child talks or makes sounds 14. Reply to child’s words or sounds 15. Wait for child’s response after making a

suggestion

16. Encourage child to do things with toys 17. Support child’s choices 18. Help child do things on his or her own 19. Verbally encourage child’s efforts 20. Offer suggestions to help child 21. Show enthusiasm about what child does 22. Explain reasons for something to child 23. Suggest activities to build on what child is

doing 24. Repeat or expand child’s words or sounds 25. Label objects or actions for child 26. Engage in pretend play with child 27. Do activities in a sequence of steps 28. Talk about characteristics of objects 29. Ask child for information

What is the KIDI?

Knowledge of Infant Development Inventory

• The theoretical base for the KIDI was the theory that parental knowledge of infant development can influence parental behavior and child rearing practices.

• The KIDI is a 58-item instrument that was designed to obtain comprehensive information on parents’ factual knowledge of parental practices, child developmental processes, and infant norms of behavior.

• The KIDI is designed to be easily accessible to persons with limited education and to be culturally neutral.

What is the PSI?

The Parenting Stress Index - Short Form• The PSI-SF consists of 36 items derived from the PSI which comprise

three scales: Parental Distress, Difficult Child Characteristics, and Dysfunctional Parent-Child Interaction.

• Designed to evaluate the magnitude of stress in the parent-child system

• The PSI-SF is commonly used as a screening and triage measure for evaluating the parenting system and identifying issues that may lead to problems in the child's or parent's behavior.

• Asking parents to complete the PSI—SF can be uncomfortable. Field Data Collectors explain that it will provide information that will help identify the sources and different types of stress that every parent can experience.

What is the HOME?• Home Observation for Measurement of the

Environment (HOME) is a descriptive profile which yields a systematic assessment of the caring environment in which the child is reared.

• The primary goal of the instrument is to measure, within a naturalistic context, the quality and quantity of stimulation and support available to a child in the home environment.

• Its focus is on the experience of the child in the home environment.

Other Resources• Visit Tracker has various recorded webinars available. New staff should watch Illinois MIECHV Training for

New Staff. https://visittrackerweb.com/ • The Governor’s Office webpage has numerous up-to-date documents posted including a MIECHV Data

Collection Manual. http://www2.illinois.gov/gov/OECD/Pages/MIECHVP.aspx • Field Data Collection brochures for each MIECHV community (contact Mary Anne Wilson for copies)• Contacts

• Teresa Kelly, Project Director, Strong Foundations Partnership- Governor's Office of Early Childhood Development 312-814-0905 teresa.m.kelly@illinois.gov

• Lesley Schwartz, Manager of Program Evaluation- Governor's Office of Early Childhood Development 312-814-4841 lesley.schwartz@illinois.gov

• Joanna Su, Community Systems & Capacity Building Manager-Governor's Office of Early Childhood Development 312-814-6741 Joanna.su@illinois.gov

• Audrey Moy, Manager of Compliance- Governor's Office of Early Childhood Development 312-814-3141 audreymoy@illinois.gov

• Peter Mulhall, Director , CPRD-University of Illinois 217-244-3231 mulhall@illinois.edu

• Mary Anne Wilson, MIECHV Research Project Specialist at CPRD, University of Illinois 217-333-3231 mawilso@uillinois.edu

• Deborah Kemmerer, MIECHV CQI Specialist at CPRD, University of Illinois 217-300-4158 dkem@uillinois.edu