research and evaluation to strengthen practice and ......research and evaluation to strengthen...
TRANSCRIPT
![Page 1: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/1.jpg)
Research and Evaluation to Strengthen Practice and Demonstrate Outcomes
Elizabeth McFarlane, PhD, MPH
Makati City, Philippines May, 2011
![Page 2: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/2.jpg)
Today’s Talk
•5 Tiered Approach to Evaluation •Prevention Science and Early Childhood •Research to Practice
•Country Reports •Hawaii’s HSP
•Expanding the Knowledge Base •NCS
![Page 3: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/3.jpg)
Five-Tiered Approach to Evaluation 1. Needs Assessment 2. Monitoring and Accountability 3. Quality Review and Program Clarification 4. Achieving Outcomes
• Establish Efficacy, then effectiveness
5. Assessing Impact • Experimental Design to Establish Causality • Replication to Establish Replicability
Program Evaluation
![Page 4: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/4.jpg)
Prevention Science
• IOM call for National Quality Measures for Pediatric Healthcare
• SRCD Social Policy Report 2011 – Global ECD •Focus on equity in ECD programs to include access and quality
![Page 5: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/5.jpg)
Prevention Science
• Early Childhood sets the life course trajectory
• Crucial Domains •Self-regulation: controlling ones emotions, behaviors, attention
•Cognitive Development: language, reasoning, problem solving
•Social Development: capacity to trust, love and resolve conflict
• Parenting mediates most of the impact of poverty
• Investment in ECD saves $$ by preventing later costs
![Page 6: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/6.jpg)
Science Points Toward 3 Types of Services
to Ensure Healthy Development
Basic health care, child care, and early learning opportunities to help all children build and sustain
strong bodies and brains.
Broadly targeted supports for children faced with major risk factors for poor development.
Specialized interventions for children and families with
complex needs.
SPECIALIZED
BROADLY TARGETED
BASIC SERVICES
Hirokazu Yoshikawa, Ph.D (ACF, 2011)
![Page 7: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/7.jpg)
Program Evaluation Research Helps Identify Effectiveness Factors
Not all programs are effective.
Effectiveness factors are key to distinguishing those programs that work from those that do not.
Hirokazu Yoshikawa, Ph.D (ACF, 2011)
![Page 8: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/8.jpg)
Ecologic Setting and Systems Levels and Cross-Cutting Quality Dimensions Britto, Yoshikawa, Boller: Social Policy Report Vol. 25 (2) 2011
![Page 9: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/9.jpg)
Solid Program Infrastructure
•Well-defined model
•Strong leadership
•Clear organizational structure
•Strong hiring and training component
•Supervision and support mechanisms for consultants
•Strategic partnerships
•Community outreach and engagement
•Clear communication
•Evaluation
•Financing 21
![Page 10: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/10.jpg)
Sample of Programs Across Counties/Hawaii’s HSP
![Page 11: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/11.jpg)
Country Program Reports
• Beijing: Uses annual statistics to monitor the success of programs to reduce infant mortality
• Shanghai: Planning a needs assessment of children’s mental health to have data for use in policy and program planning
• Hong Kong: GIS systems to identify injury patterns. Home-based intervention developed to address household risks.
• Hong Kong: Representative Survey of Child and Adolescent Health
• Philippines: Needs assessment of children with autism transitioning to adulthood
• Singapore: Data monitoring to assure impact
![Page 12: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/12.jpg)
Research to Practice Partnership: Hawaii’s Healthy Start Program and JHU
Partnership since 1994
Several Studies: RCT – RWJ, Annie E. Casey, Packard Foundation, MCHB, HDOH RCT – NIMH, NICHD Fidelity Study – CDC EBHV - ACF
Ongoing National Interest in Home Visiting U.S. Congress Patient Protection and Affordable Care Act 2010, provides a $1.5 billion investment in state expansion of evidence-based early home visiting.
![Page 13: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/13.jpg)
Rationale for RCT of Hawaii’s Healthy Start Program
Strong design + program taken to scale
Commitment to evaluation and dissemination
National Interest “In 1990 the US Advisory Board on Child Abuse
described the program as “clearly the star” of US home visitation programs…. A rigorous evaluation is needed … The ideal way of determining program effectiveness would be a randomized controlled trial with multiple outcome measures.”
Institute of Medicine, 1994
![Page 14: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/14.jpg)
Hawaii’s Healthy Start Program
• Goals •To improve family functioning •To promote child health and development •To prevent child abuse and neglect
• Components •Population-based screening •Home visiting
![Page 15: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/15.jpg)
Study Methods
•Sample Cross-sectional study: 6545 births Randomized trial: 684 at-risk families
•Follow-up •Blinded, independent fieldwork staff
•Feedback
•To inform quality improvement
![Page 16: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/16.jpg)
Program Impact
• Decreased partner violence/injury*
• Increased use of nonviolent discipline
• Increased access to a medical home
• Increased continuity of care
• Improved parent-child interaction*
• Improved child development scores*
* At some HSP sites
![Page 17: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/17.jpg)
What We Know from Meta-Analytic Studies of Home Visiting : HV can be effective, but effects are small.
Domain ES Cognitive Development (41 studies) 18* Socio-emotional Development (24) .10* CAN Prevention (7) .32 CAN – Potential Abuse (13) .24* Parenting Stress (4) .21 Parenting Behavior (37) .14* Parenting Attitudes (15) .10* Maternal Education (5) .13* Maternal Employment (7) .02 Public Assistance (3) -.04
ES Key Small .20
Medium .50
Large .80
*p<.05
![Page 18: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/18.jpg)
What We Know from RCTs of Home Visiting
•Programs target the right family outcomes in the right families at the right time.
•Program effects are modest.
•Actual services depart from models.
•Implementation systems weaknesses were the reason for these departure.
•Discrete protocols can improve outcomes.
•Challenges are the same for primary care and home visiting.
![Page 19: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/19.jpg)
Enhancing Hawaii’s Statewide Home Visiting Program to Improve Fidelity and Effectiveness
• Improved Fidelity & Effectiveness
• Cultural Relevancy
• Continuous Quality Improvement System
![Page 20: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/20.jpg)
Evaluation Approach
Conceptual Framework for Implementation Fidelity (Carroll C. et al. (2007)
20
Intervention Fidelity
Coverage Duration & Frequency
Details of Content
Outcomes
Potential Moderators Model complexity & clarity Implementation strategies Participant responsiveness
Evaluation of Implementation Fidelity
Evaluation
Component analysis to identify
“essential” components
![Page 21: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/21.jpg)
Evaluation Approach (Continued)
Community-Based Participatory Research and Quality Improvement Methodologies Used To Improve:
• Home Visiting Infrastructure (e.g. curricula, supervision) • Practice (e.g. family recruitment, visit frequency) • Outcomes (e.g. family functioning, prevention of CAN)
21
![Page 22: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/22.jpg)
The Science to Service Journey…
• What is known is not what is adopted in practice
• Implementation is a process, not an event.
• Adaptation must be evidence-based.
• Without monitoring, fidelity declines over time.
• To achieve intended outcomes, we must: •Adopt efficacious models •Adapt them carefully and cautiously •Achieve fidelity through training and monitoring •Test our results
![Page 23: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/23.jpg)
Expanding the Knowledge Base
![Page 24: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/24.jpg)
The National Children’s Study
• Largest long-term study of children’s health and development ever to be conducted in the U.S.
• Longitudinal study of children, their families, and their environment (before birth through age 21)
• Approximately 100,000 children enables study of important but less common outcomes
![Page 25: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/25.jpg)
* Reappointed 2001 and 2003
Rationale for the National Children’s Study
From The President’s Task Force on Environmental Health and Safety Risks to Children, 2000*
• Compared to adults, children are especially vulnerable to environmental exposures – metabolism, behavior
• Exposures to some agents demonstrate potential for serious developmental effects – lead, prenatal alcohol
• Current known exposures of high frequency – pesticides, violence, media
• Numerous high burden conditions with suspected environmental contribution – learning disabilities, autism, diabetes, asthma, birth defects, premature birth
• Existing research too limited in size and scope to answer the questions
• Life-course (longitudinal) design needed to correctly link with multiple exposures and multiple outcomes
![Page 26: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/26.jpg)
Converging factors
• Numerous experiences of adverse effects of environmental exposures on children’s health and development
• Many identified environmental exposures with concerns about toxic and other adverse effects in children
• Diseases and developmental conditions with possible environmental causes or contributions
![Page 27: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/27.jpg)
Priority Environmental Exposures
• Physical environment: housing, neighborhoods and communities, climate, radiation…
• Chemical exposures: air, water, soil, food, dust, industrial products, pharmaceuticals… • complex ubiquitous low-level exposures • unique exposures (special sub-studies)
• Biological environment: intrauterine, infection, nutrition; inflammatory and metabolic response…
• Genetics: genetic components of disease; effects of environmental exposures on gene expression…
• Psychosocial milieu: influence of family, socio-economics, community, stress…
![Page 28: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/28.jpg)
Examples of Hypothesis-defining questions
• How is asthma incidence and severity influenced by the interaction of early life infection and air quality?
• Are assisted reproductive technologies (ART) at increased risk of fetal growth restriction, birth defects, and developmental disabilities?
• Does impaired maternal glucose metabolism during pregnancy cause obesity in children?
• How does high level exposure to media content in infancy affect development and behavior in children?
• Does pre-and post-natal exposure to endocrine-active environmental agents alter age at onset, duration, and completion of puberty?
![Page 29: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/29.jpg)
Priority Health Exposures, Outcomes
Priority Exposures
Examples
Physical Environment Housing quality, neighborhood
Chemical Exposures Pesticides, phthalates, heavy metals
Biologic Environment Infectious agents, endotoxins, diet
Genetics Interaction between genes and environment
Psychosocial milieu
Family structure, socio-economic status, parenting style, social networks, exposure to media and violence
Priority Health Outcomes
Examples
Pregnancy Outcomes Preterm, Birth defects
Neurodevelopment & Behavior
Autism, learning disabilities, schizophrenia, conduct and behavior problems
Injury Head trauma, Injuries requiring hospitalizations
Asthma Asthma incidence and exacerbation
Obesity & Physical Development
Obesity, diabetes, altered puberty
![Page 30: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/30.jpg)
National Children’s Study Sample
105 Locations
Selection of neighborhoods
All or a sample of households within neighborhoods
All eligible women in the household
~4 million births in 3,141 counties
All Births in the Nation
Sample of Study Locations
Sample of Study Segments
Study Households
Study Women
![Page 31: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/31.jpg)
![Page 32: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/32.jpg)
Study Participation
• Recruit from pre-conception/early pregnancy • Diverse populations: ethnicity, SES, family structure • Minimum of 16 in-person home and clinic-based visits
(pre-conception through 21) • Contact by telephone, computer, mail-in questionnaires
every 3 months until 5 years old; then annually • Environmental (air, water, soil, dust) samples from
child’s environment (home, school, day care) • Psychosocial, demographic, neurodevelopmental,
neighborhood and contextual data • Biological samples from mother, father, child
![Page 33: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/33.jpg)
What will the NCS Mean to our Children’s Well-Being?
• Identification of environmental factors which cause or contribute to health, development and behavior problems • Asthma, injury, obesity, autism, ADHD, prematurity…
• Understanding the biology and genetics of health, development and behavior
• Evidence-based information on which to base decisions about practice and policy regarding children’s physical and mental health
• Economic benefits: cost avoidance
• Resource for future research
![Page 34: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/34.jpg)
So, the NCS will Provide…
• The answer to concerns about known exposures during childhood to potential toxicants
• The power to determine absence of effects or benefit of exposures to various products important for our society
• Causal factors for a number of diseases and conditions of children with suspected environmental causes
• How multiple causes interact to result in multiple outcomes
• Large sample size required to apply knowledge of the human genome to understand multi-factorial genetic conditions
• Identification of early life factors that contribute to many adult conditions
• A national resource to answer future questions by using stored biological and environmental samples and the extensive data for decades to come
![Page 35: Research and Evaluation to Strengthen Practice and ......Research and Evaluation to Strengthen Practice and Demonstrate Outcomes Elizabeth McFarlane, PhD, MPH Makati City, Philippines](https://reader033.vdocuments.mx/reader033/viewer/2022041514/5e29dddcd0f0d04a08394966/html5/thumbnails/35.jpg)
Thank You! Maraming Salamat!
Questions?