Evaluating Complex Systems of CareA Logic Model Approach to EHDI Program Evaluation

Sherry Spence, MA

Oregon Department of Human Services

Health Services, Office of Family Health

National EHDI ConferenceAtlanta, Georgia, March 3, 2005

Supported in part by funding from the Maternal and Child Health Bureau (Title V, Social Security Act), Health Resources and Services Administration, and by the Centers for

Disease Control and Prevention, US Department of Health and Human Services

Opinions stated do not represent those of CDC, HRSA, or USDHHS

Acknowledgements EHDI Staff at CDC and HRSA EHDI Data Committee Oregon’s EHDI Team

• Claudia Bingham

• Al Ferro

• David Laszlo

• Maureen McNamer

• Amy Rosenthal

Acknowledgements

FamilyNet Executive Sponsors• Katherine Bradley

• Lorriane Duncan Office of Family Health

Epidemiologists and Research Analysts

FamilyNet Project Teams and . . .

And Oregon’s Families

Foreword

“If you torture data sufficiently, it will confess to almost anything.”• Fred Menger (1937- ), chemistry

professor• From A Word A Day (AWAD) -

wsmith@wordsmith.org

Foreword

Corollary: “Sometimes you can coerce data by threatening deletion.”• Arthur Epstein, professor of chemistry

and physics (from daughter, Melissa, responding in AWADmail in response to the Menger quote)

Foreword

Conclusion: “There are lies, damn lies, and statistics.”• Mark Twain (1835-1910), author

and humorist

What is our goal for today?

Disprove the Twain/Menger-Epstein hypothesis

Working hypothesis:• It is possible, without false

testimony, torture, or coercion, to develop an evaluation plan using our program data.

What is our goal, really?

What is our goal for today? Define a logical framework for

evaluating EHDI programs Discuss whether our logic model

approach avoids the Twain/ Menger-Epstein condition.

Figure out where our model fits in identifying newborn hearing screening as a best practice.

What are we currently evaluating?Logical framework for EHDI evaluation

HospitalsDiagnostic

FacilitiesEarly Intervention

Facilities

Parent

DHS

Healthcare Provider

Within 10Days of

Screening, Testing or

Referral

6 weeksafter

“Refer”

2 weeks after

Dx Hearing loss if noEI referral

ChildIs

6 weeks of age

Parent

Healthcare Provider

Local Public Health

Parent

Healthcare Provider

Local Public Health

Reminder letters / program contact only when child has not completed the desired EHDI process by the desired time.

Participating facilities, HCP and LPH able to return updated contact information

Monthly reports, program statistics back to facilitiesDemographics, screening, testing and referral information to DHS

DHS Response When:

Screening Facilities

What are we evaluating?

Input

Output

Intermediate Outcome

Intermediate Outcome

Resources & needs

Protocol measure

Performance Measure

Program Measure

EHDI staffing & funding

Report & follow-up completeness &timeliness

Diagnosis, intervention, & risk follow-up

Child served can communicate & learn

What are we trying to prove? Success in achieving EHDI goals

• Early screening of everyone

• Early identification of some

• Early preventive or treatment intervention

• Acquisition of language and communication skills

What else do we need to show?

Success in achieving EHDI goals

Success in achieving health care goals & optimizing resource use

Success in all children achieving their full health and development potential

What are the components of an EHDI logic model?

Input

Output

Intermediate Outcome

High Level Outcome

Resources & needs

Performance measure

Program Measure

Population Measure

EHDI staffing & funding

Screen & Dx completeness & timeliness; follow-up

Children served can communicate & learn

All children communicate & are ready for school

What are the components of an EHDI logic model?

Children interacting and ready to learn

Language and communication skills

Intervention to assure skill acquisition

Early and complete identification and risk monitoring

High level outcomes - population attributes

Intermediate outcomes - client/patient attributes

Outputs - program activities and interventions

What data do we have?

Input

Output

Intermediate Outcome

High Level Outcome

Resources & needs

Performance measure

Program Measure

Population Measure

4.0 FTE, 2 Grants

93% screened 90% screened and linked 70% diagnosed within 3 mos. 7 of 49 in EI (per reports)

NA

NA

What is the approximate timing of this monitoring? All children ready to

interact and learn Language and

communication skills Intervention to assure

skill acquisition Early and complete

identification and risk monitoring

4-5 years after birth

12-36 months after birth

3-6 months after birth

1-3 months after birth

So we won’t know for 4-5 years if this was a good idea?

How can we use this logic model in program evaluation? Conduct research

Conduct research

Monitor program outcomes

Monitor program protocols

4-5 years after birth?

12-36 months after birth?

3-6 months after birth

1-3 months after birth

Will system integration help?

Earlier screening and identification More complete screening and

identification More timely follow-up Less loss to follow-up

Why do we think this is so? About 40% of hearing loss occurs with

other conditions or syndromes, resulting in additional disabilities (Gallaudet Research Institute, 2003)• Additional issues increase likelihood of other

public health program involvement. In Oregon, about 50% of women giving

birth are enrolled in WIC and will enroll their babies• The WIC program keeps track of its highly

mobile population.

How does the logic model work for systems?

Input

Output

Intermediate Outcome

High Level Outcome

Resources & needs

Protocol measure

Performance measure

Performance result

EHDI system require-ments

Strategic plan, design, system works

Facilitation of diagnoses, referrals, and follow-up

Increases number who communicate & are ready to learn

What are our intermediate system outcomes? COORDINATION

CONTINUITY

EFFICIENCY

TIMELINESS

ACCURACY & APPROPRIATENESS

How does this fit with the logic model?

Population success

Program success

Process success

Program Logic Model

System Logic Model

Conclusions

One purpose of a system of services is to assure the success of the individual programs and processes it involves.

One purpose of a data system is to facilitate and enhance the success of the programs that use it.

What do we have so far?

Solid framework for measuring program processes

Solid framework for measuring system processes

So what?

What’s the key to all this?

The logic model works best if the program intervention is an established, evidence-based best practice.

What best practice evidence to we need? Gold standard Controlled

studies by programs

Expert testimony by professionals

Repeated measures and testimony

Immoral or unethical Good evidence from

which we can generalize?

Good evidence or bias?

Eliminates bias and demonstrates generalizability

What evidence do we have?

Prospective and retrospective studies by programs and researchers?

Expert testimony by speech, hearing, and language professionals?

Repeated measures of all kinds?

Thank you.

Thanks to

Amy Rosenthal, Oregon EHDI Coordinator for her participation and to

Scott Grosse, PhD, NCBDDD/CDC, for his participation and for noting an error in slide

32.